Commit 7012479b authored by Daniela Daniel's avatar Daniela Daniel
Browse files

first commit

parent f989acee
*.pyc
# Ignore pycharm/idea directory
.idea/
# Ignore any nfs lock stuff
.nfs*
# Don't include the virtualenv/pyenve and build files
.pyenv/
.python-version
.cache
env/
# Don't include CMS media files
# Use rsync to transfer if needed
media/filer*
media/uploads
# Don't include test sqlite db
database.sqlite
# Ignore stuff produced by manage.py collectstatic
static/admin/
static/cms/
static/cmsplugin_filer_folder/
static/cmsplugin_zinnia/
static/djangocms_admin_style/
static/djangocms_text_ckeditor/
static/filer/
static/zinnia/
static/biohpc_sbatch
static/cmsplugin_xdmod
static/treebeard
staticfiles/
ftp_site/settings.py
# Don't include secret / password / local settings
ftp_site/settings_local.py
certs/
# Don't include vagrant VM directory
.vagrant/
# Don't include celery beat schedule
celerybeat-schedule
# Don't keep log files in git
*.log
log/*.log
log/*.pid
# Bower
*bower_components
node_modules/
yarn.lock
\ No newline at end of file
This diff is collapsed.
Copyright ©2020, University of Texas Southwestern Medical Center. All rights reserved.
Contributors: David C. Trudgian, Long Lu, Daniela F. Daniel
Department: BioHPC, Lyda Hill Department of Bioinformatics
This software is licensed under GPL v3.0
(https://www.gnu.org/licenses/gpl-3.0.en.html)
For an alternative license, please contact the Office for Technology Development
at University of Texas Southwestern Medical Center, Dallas, TX
(TechnologyDevelopment@utsouthwestern.edu)
\ No newline at end of file
# FTP Setup Guide
### Overview
This FTP application (Flash FTP) relies on `vsftpd` and `pam_mysql` so that FTP guest users
are not required to be LDAP system users. The credentials for an FTP guest user
reside in a MySQL table and `pam_mysql` handles taking the FTP username/password,
hashing it and comparing it to the username/password fields in the database
table, thus allowing or denying guest users access to the FTP service. Users
that wish to share files with a guest are able to login into the system (i.e.,
authenticate against LDAP) and send invitations to guests users by e-mail.
This documentation assumes that the Django app resides in `/devel/ftp_project`
and the FTP directory is `/project/ftp_public`.
![Service Diagram](./service_diagram.png)
### Components
- Python 3.6.8
- Django 2.2 LTS
- MySQL
- LDAP
- Folders, file permissions and mounts
- IP Tables/Firewall Daemon
### Install EPEL and build packages
```sh
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum -y groupinstall "Development Tools" "Development Libraries"
```
### MySQL
Edit the file `/etc/yum.repos.d/mysql-community.repo`:
```sh
# Enable to use MySQL 5.7
[mysql57-community]
name=MySQL 5.7 Community Server
baseurl=http://repo.mysql.com/yum/mysql-5.7-community/el/7/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql
```
Install MySQL:
```sh
yum repolist enabled | grep mysql
yum install mysql-community-server
systemctl start mysqld.service
systemctl enable mysqld.service
```
### Create the database
```sh
mysql -u root
mysql> CREATE DATABASE vsftpd;
mysql> exit;
mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql
```
### FTP
Create user `vsftpd` and configure `vsftpd`:
```ssh
useradd -G users -s /bin/false -d /home/vsftpd vsftpd
passwd vsftpd
yum install -y vsftpd
cp -p /etc/vsftpd/vsftpd.conf /etc/vsftpd/vsftpd.conf-backup
cp -p provision/vsftpd/vsftpd.conf /etc/vsftpd/vsftpd.conf
chown root.root /etc/vsftpd/vsftpd.conf
mkdir -p /project/ftp_public/
mkdir -p /project/vsftpd_user_conf
chmod 755 /project/ftp_public/
chown <vsftpd_service>:<vsftpd_service> /project/ftp_public/
systemctl start vsftpd
systemctl enable vsftpd
```
### User directories
Create an FTP directory for each FTP user under `/project/ftp_public/`:
```sh
mkdir <user>
chmod 700 <user>
chown <vsftpd_service>:<vsftpd_service> <user>
chmod g+s <user>
setfacl -R -m u:<user>:rwx <user>
setfacl -R -m u:vsftpd-guest:rwx <user>
setfacl -R -m d:u:vsftpd-guest:rwx <user>
setfacl -R -m d:u:<user>:rwx <user>
```
### Create user `vsftpd` in the database
```sh
mysql> CREATE USER 'vsftpd'@'localhost' IDENTIFIED BY 'vsftpd';
mysql> GRANT ALL PRIVILEGES ON vsftpd.* TO 'vsftpd'@'localhost';
```
### PAM MySQL
Download and install `pam_mysql`. Check if there's a library called _pam_mysql.so_ in folder `/usr/lib64/security/`.
```sh
cp /etc/pam.d/vsftpd vsftpd.orig
cp provision/vsftpd/vsftpd_pam /etc/pam.d/vsftpd
chown root:root /etc/pam.d/vsftpd
```
Download mirrors: https://centos.pkgs.org/7/cheese-x86_64/pam_mysql-0.7-0.21.rc1.el7.x86_64.rpm.html
### LDAP client
```sh
yum install openldap-clients nss-pam-ldapd pam_ldap
authconfig --enableldap --enableldapauth --ldapserver="<ldap_server>" --ldapbasedn="<base_dn>" --update
authconfig --enableforcelegacy --update
```
You might need to edit `/etc/nslcd.conf` to provide cetificate info. Test the client:
```sh
systemctl restart nslcd
getent passwd
```
### IP Tables
```sh
yum install iptables-services
iptables --flush INPUT
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 20 -j ACCEPT
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 3306 -j ACCEPT
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 21 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --sport 1024:65535 --dport 20:65535 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -j DROP
service iptables save
systemctl enable iptables
```
### Python 3
Install Python 3 and set up a virtual environment for user _<vsftpd_service>_. Then become _<vsftpd_service>_ user:
```sh
pip install --upgrade pip
pip install -r requirements.txt
pip install gunicorn
```
### Deploy the Django database
```sh
python manage.py makemigrations
python manage.py migrate
```
Create the authentication view for library `pam_mysql.so`:
```sh
mysql> CREATE VIEW account_table AS SELECT username,password from ftp_manager_secondaryaccount WHERE expiration_time>now() UNION SELECT username,password FROM ftp_manager_account WHERE expiration_time>now();
```
__Edit `settings.py` to suit your needs (LDAP and FTP and e-mail settings):__
```
# ----------------------------------------------------------------------------
# biohpc_accounts app
# ----------------------------------------------------------------------------
# LDAP URI, user and password for BioHPC Directory (read/write)
BIOHPC_ACCOUNTS_LDAP_URI = "ldap://localhost"
BIOHPC_ACCOUNTS_LDAP_USER = ''
BIOHPC_ACCOUNTS_LDAP_PASSWORD = ''
# Default primary GID for newly created users
BIOHPC_ACCOUNTS_DEFAULT_GID = 9999
# Samba Domain Name
BIOHPC_ACCOUNTS_SAMBA_DOMAIN = ''
# Samba Domain SID
BIOHPC_ACCOUNTS_SAMBA_SID = ''
# LDAP Base DN to check for users/groups anywhere in the tree
BIOHPC_ACCOUNTS_BASE_DN = ''
# LDAP suffix for new users added to directory, but not enabled for services
# Have portal access only
BIOHPC_ACCOUNTS_NEW_USER_SUFFIX = ''
# LDAP suffix for active users in directory
BIOHPC_ACCOUNTS_ACTIVE_USER_SUFFIX = ''
# LDAP Suffix for group search
BIOHPC_ACCOUNTS_GROUP_SUFFIX = ''
# Permitted domains for users email at registration
BIOHPC_ACCOUNTS_GOOD_EMAIL_DOMAINS = ['', '']
# ----------------------------------------------------------------------------
# ftp_manager app
# ----------------------------------------------------------------------------
FTP_SECRETE = ''
FROM_EMAIL = ''
```
### Run locally
Set `DEBUG=True` and test the application:
```sh
python manage.py runserver
```
### Static files and Apache
During development, if you use `django.contrib.staticfiles`, this will be done automatically by runserver when `DEBUG` is set to `True`. For production, run `collectstatic` to copy all the static files into `STATIC_ROOT`:
```sh
python manage.py collectstatic
```
Have `httpd` serve static files:
```sh
yum install httpd
cp provision/httpd/apache.conf /etc/httpd/conf.d/apache.conf
systemctl start httpd
systemctl enable httpd
```
In `settings.py`, set `DEBUG` and `TEMPLATE_DEBUG` to `False`.
### Final check
```
python manage.py check --deploy
```
### Run FTP application
```sh
gunicorn ftp_site.wsgi:application
```
## Contributing to Flash FTP
We are open to contributions from external developers. We advise you to open an issue that describes the contribution and then submit a pull request.
For additional details, refer to the LICENSE file contained in this repository.
This biohpc_accounts module incorporates code from django_registrations...
https://github.com/nathanborror/django-registration
.. which was released under the following licens:
Copyright (c) 2007-2010, James Bennett
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided
with the distribution.
* Neither the name of the author nor the names of other
contributors may be used to endorse or promote products derived
from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
Routines to work with the BioHPC LDAP Directory
Replaces use of django_ldapdb
"""
import logging
import ldap
from django.conf import settings
from django.core.mail import send_mail
LOGGER = logging.getLogger('django')
def get_ldap_mail(user):
'''query LDAP for user mail'''
con = ldap.initialize(settings.BIOHPC_ACCOUNTS_LDAP_URI)
con.simple_bind_s(settings.BIOHPC_ACCOUNTS_LDAP_USER,
settings.BIOHPC_ACCOUNTS_LDAP_PASSWORD)
try:
result = con.search_s(settings.BIOHPC_ACCOUNTS_BASE_DN,
ldap.SCOPE_SUBTREE, "uid=" + user, ['mail'])
except ldap.LDAPError as ldap_error:
LOGGER.error('ldap query error: %s', str(ldap_error))
return False
con.unbind_s()
if len(result) == 0:
LOGGER.error('ldap mail not found')
return False
if len(result) > 1:
LOGGER.error('user has multiple emails')
return False
return result[0][1]['mail']
def send_ldap_mail(subject, message, from_email, user):
'''send mail to user'''
send_mail(subject, message, from_email,
get_ldap_mail(user), fail_silently=False)
{% load sekizai_tags static %}
{% load widget_tweaks %}
{% addtoblock "css" %}
<link href="{% static 'bower_components/bootstrap/dist/css/bootstrap.min.css' %}" media="screen" rel="stylesheet" type="text/css"/>
<link href="{% static 'bower_components/font-awesome/css/font-awesome.min.css' %}" rel="stylesheet" type="text/css">
<link href="{% static 'css/login.css' %}" rel="stylesheet" type="text/css">
{% endaddtoblock %}
<!doctype html>
<html>
<head>
<title>BioHPC FTP:: BioHPC Login</title>
{% render_block "css" %}
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
</head>
<body>
<div class="site-wrapper">
<div class="site-wrapper-inner">
<div class="cover-container">
<div class="inner cover">
<div class="panel panel-default">
<div class="panel-heading">
<img src="{% static 'images/logos/biohpc_logo_100pxh.png' %}"/>
</div>
<div class="panel-body">
<h4 class="cover-heading">Login to BioHPC Flash FTP (<strong class="text-primary">EXTERNAL</strong>)</h4>
<p class="cover-lead">A UTSW BioHPC account is required to access the FTP service on BioHPC</p>
<p><strong>Files stored and transferred MUST NOT contain PHI or other privacy sensitive information without prior approval.</strong></p>
</div>
</div>
<div class="panel panel-primary">
<div class="panel-heading">
<h3 class="panel-title">BioHPC Login</h3>
</div>
<div class="panel-body">
{% if form.errors %}
<div class="alert alert-danger" role="alert">Your username and password didn't match. Please
try again.
</div>
{% endif %}
<form class="form" method="POST" autocomplete="off" action="{% url 'login' %}">
{% csrf_token %}
<div class="row">
<div class="col-md-8 col-md-offset-2">
<div class="form-group">
<div class="input-group">
<label class="sr-only" for="username">BioHPC Username</label>
<div class="input-group-addon"><i class="glyphicon glyphicon-user"></i>
</div>
{{ form.username|add_class:"form-control"|append_attr:"placeholder:BioHPC Username"|append_attr:"autocomplete:off" }}
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-8 col-md-offset-2">
<div class="form-group">
<div class="input-group">
<label class="sr-only" for="password">Password</label>
<div class="input-group-addon"><i class="glyphicon glyphicon-lock"></i>
</div>
{{ form.password|add_class:"form-control"|append_attr:"placeholder:Password" }}
</div>
</div>
</div>
</div>
<input type="hidden" name="next" value="/{{ next }}"/>
<div class="row margin-top-10">
<div class="col-md-8 col-md-offset-2">
<button class="btn btn-lg btn-primary btn-block" type="submit" value="Login"><i
class="glyphicon glyphicon-log-in"></i> Login
</button>
</div>
</div>
</form>
<div class="row margin-top-10">
<div class="col-md-4 col-md-offset-2">
<a class="btn btn-danger btn-block" href="https://portal.biohpc.swmed.edu/accounts/password-reset"><i
class="glyphicon glyphicon-lock"></i> Reset Password </a>
</div>
<div class="col-md-4">
<a class="btn btn-success btn-block" href="https://portal.biohpc.swmed.edu/accounts/register"><i
class="glyphicon glyphicon-plus-sign"></i> New
account</a>
</div>
</div>
<div class="row margin-top-10">
<div class="col-md-8 col-md-offset-2">
</div>
</div>
</div>
<p class="cover-lead"><strong>Problems logging in?</strong><br>
<p>Accounts are available for users in participating departments. If your department is
interested in becoming a participating department, able to access BioHPC resources,
please contact:</p>
<i class="glyphicon glyphicon-envelope"></i> <a href="mailto: biohpc-help@utsouthwestern.edu">biohpc-help@utsouthwestern.edu</a><br/>
</p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>
"""
biohpc_accounts test cases.
"""
#from django.test import TestCase
# Create your tests here.
"""
ftp_manager URL configuration
"""
from django.urls import path
from django.contrib.auth.views import LoginView, LogoutView
urlpatterns = [
path('login/', LoginView.as_view(template_name='biohpc_accounts/login.html'), name="login"),
path('logout/', LogoutView.as_view(next_page='/'), name="logout"),
]
"""
Django admin site.
"""
# from django.contrib import admin
# Register your models here.
"""
Django forms.
"""
from django import forms
class SendInvitation(forms.Form):
'''
Basic email form.
'''
email = forms.EmailField(label='Email', max_length=100, required=True)
user_note = forms.CharField(label='Note', max_length=100)
message = forms.CharField(label='Message', widget=forms.Textarea)
"""
Flash FTP application main controller.
"""
import os
import smtplib
import hashlib
import datetime
import logging
import subprocess
from django.utils import timezone
from django.utils.crypto import get_random_string
from django.core.mail import send_mail
from django.core.exceptions import ObjectDoesNotExist
from django.template.loader import render_to_string
from ftp_manager.models import Account, SecondaryAccount
from ftp_site.settings import FTP_DIRECTORY, VSFTPD_USER_CONF_DIR,\
VSFTPD_USER_CONF_SKEL, VALID_DAYS, FTP_HOSTNAME, FTP_SECRETE, FROM_EMAIL,\
FTP_IPADDRESS, MAX_GUEST_COUNT, FTP_SERVICE_USER
LOGGER = logging.getLogger('django')
# BioHPC user account methods
def get_user_info(username):
''' returns user account info: username, password and expiration date'''
LOGGER.info('%s logged in', username)
if Account.objects.filter(username=username).count() == 0:
account = get_new_user_password(username)
return account
account = Account.objects.get(username=username)
return account
def disable_ftp_account(username):
'''disables user account by username by setting expiration time to now'''
account = Account.objects.get(username=username)
if account.expiration_time > timezone.now():
account.disable_user_password()
LOGGER.info('%s account disabled', username)
return True
return False
def get_new_user_password(username):
'''
creates user account by username, if new user logs in for the first time,
creates new directory and database entry for user
'''
password, hash_pass = generate_pass_md5()
expiration_time = timezone.now() + datetime.timedelta(days=VALID_DAYS)
<