Toolchain/Dependency Upgrade (#418)

* Upgrade to heroku-20 stack

* Move some gulp deps to dev rather than prod

* npm upgrade

* Fix audit time check in asset audit test

* Attempt at parallelising tests where possible

* Add basic calendar button test

Mainly to pickup on FullCalendar loading errors

* Upgrade python deps

* Tends to help if I push valid yaml

* You valid now?

* Fix whoops in requirements.txt

* Change python ver

* Define service in coveralls task

* Run parallelised RIGS tests as one matrix job

* Update python version in tests

* Cache python dependencies

Should majorly speedup parallelillelelised testing

* Purge old vagrant config

* No Ruby compass bodge, no need for rubocop!

* Purge old .idea config

* Switch to gh-a artifact uploading instead of imgur 'hack'

For test failure screenshots. Happy now @mattysmith22? ;p

* Oops, remove unused import

* Exclude tests from the coverage stats

Seems to be artifically deflating our stats

* Refactor asset audit tests with better selectors

Also fixed a silly title error with the modal

* Add title checking to the slightly insane assets test

* Fix unauth test to not just immediately pass out

* Upload failure screenshots as individual artifacts not a zip

Turns out I can't unzip things from my phone, which is a pain

* Should fix asset test on CI

* What about this?

* What about this?

Swear I spend my life jiggerypokerying the damn test suite...

* Does this help the coverage be less weird?

* Revert "Does this help the coverage be less weird?"

This reverts commit 39ab9df836.

* Use pytest as our test runner for better parallelism

Also rewrote some asset tests to be in the pytest style. May do some more. Some warnings cleaned up in the process.

* Bah, codestyle

* Oops, remove obsolete if check

* Fix screenshot uploading on CI (again)

* Try this way of parallel coverage

* Add codeclimate maintainability badge

* Remove some unused gulp dependencies

* Run asset building serverside

* Still helps if I commit valid YAML

* See below

* Different approach to CI dependencies

* Exclude node_modules from codestyle

* Does this work?

* Parallel parallel builds were giving me a headache, try this

* Update codeclimate settings, purge some config files

* Well the YAML was *syntactically* valid....

* Switch back to old coveralls method

* Fix codeclimate config, mark 2

* Attempt to bodge asset test

* Oops, again

Probably bedtime..

* Might fix heroku building

* Attempt #2 at fixing heroku

* Belt and braces approach to coverage

* Github, you need a Actions YAML validator!

* Might fix actions?

* Try ignoring some third party deprecation warnings

* Another go at making coverage show up

* Some template cleanup

* Minor python cleanup

* Import optimisation

* Revert "Minor python cleanup"

This reverts commit 6a4620a2e5.

* Add format arg to coverage command

* Ignore test directories from Heroku slug

* Maybe this works to purge deps postbuild

* Bunch of test refactoring

* Restore signals import, screw you import optimisation

* Further template refactoring

* Add support for running tests with geckodriver, do this on CI

* Screw you codestyle

* Disable firefox tests for now

That was way more errors than I expected

* Run cleanup script from the right location

* Plausibly fix tests

* Helps if I don't delete the pipeline folder prior to collectstatic

* Enable whitenoise

* Can I delete pipeline here?

* Allow seconds difference in assert_times_equal

* Disable codeclimate

* Remove not working rm command

* Maybe this fixes coverage?

* Try different coverage reporter

* Fix search_help to need login

* Made versioning magic a bit less expansive

We have more apps than I thought...

* Fix IDI0T error in Assets URLS

* Refactor 'no access to unauthed' test to cover all of PyRIGS

* Add RAs/Checklists to sample data generator

* Fix some HTML errors in templates

Which apparently only Django's HTML parser cares about, browsers DGAF...

* Port title test to project level

* Fix more HTML

* Fix cable type detail

PyRIGS/decorators.py

@@ -1,6 +1,7 @@
+from django.conf import settings
 from django.contrib.auth import REDIRECT_FIELD_NAME
-from django.shortcuts import render
 from django.http import HttpResponseRedirect
+from django.shortcuts import render
 from django.urls import reverse
 
 from RIGS import models
@@ -15,11 +16,7 @@ def get_oembed(login_url, request, oembed_view, kwargs):
     return resp
 
 
-def has_oembed(oembed_view, login_url=None):
-    if not login_url:
-        from django.conf import settings
-        login_url = settings.LOGIN_URL
-
+def has_oembed(oembed_view, login_url=settings.LOGIN_URL):
     def _dec(view_func):
         def _checklogin(request, *args, **kwargs):
             if request.user.is_authenticated:

PyRIGS/settings.py

@@ -8,11 +8,12 @@ For the full list of settings and their values, see
 https://docs.djangoproject.com/en/1.7/ref/settings/
 """
 
+import datetime
 # Build paths inside the project like this: os.path.join(BASE_DIR, ...)
 import os
-import raven
 import secrets
-import datetime
+
+import raven
 from envparse import env
 
 BASE_DIR = os.path.dirname(os.path.dirname(__file__))
@@ -77,6 +78,7 @@ INSTALLED_APPS = (
 MIDDLEWARE = (
     'raven.contrib.django.raven_compat.middleware.SentryResponseErrorIdMiddleware',
     'django.middleware.security.SecurityMiddleware',
+    'whitenoise.middleware.WhiteNoiseMiddleware',
     'debug_toolbar.middleware.DebugToolbarMiddleware',
     'reversion.middleware.RevisionMiddleware',
     'django.contrib.sessions.middleware.SessionMiddleware',
@@ -90,6 +92,7 @@ MIDDLEWARE = (
 ROOT_URLCONF = 'PyRIGS.urls'
 
 WSGI_APPLICATION = 'PyRIGS.wsgi.application'
+STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
 
 # Database
 # https://docs.djangoproject.com/en/1.7/ref/settings/#databases
@@ -235,6 +238,9 @@ STATIC_ROOT = os.path.join(BASE_DIR, 'static/')
 STATIC_DIRS = (
     os.path.join(BASE_DIR, 'static/')
 )
+STATICFILES_DIRS = [
+    os.path.join(BASE_DIR, 'pipeline/built_assets/'),
+]
 
 TEMPLATES = [
     {
@@ -263,6 +269,3 @@ USE_GRAVATAR = True
 
 TERMS_OF_HIRE_URL = "http://www.nottinghamtec.co.uk/terms.pdf"
 AUTHORISATION_NOTIFICATION_ADDRESS = 'productions@nottinghamtec.co.uk'
-
-IMGUR_UPLOAD_CLIENT_ID = env('IMGUR_UPLOAD_CLIENT_ID', default="")
-IMGUR_UPLOAD_CLIENT_SECRET = env('IMGUR_UPLOAD_CLIENT_SECRET', default="")

PyRIGS/tests/base.py

@@ -1,16 +1,17 @@
+import os
+import pathlib
+import sys
+from datetime import datetime
+
+import pytz
+from django.conf import settings
 from django.test import LiveServerTestCase
 from selenium import webdriver
+from selenium.webdriver.support.wait import WebDriverWait
+
 from RIGS import models as rigsmodels
 from . import pages
-import os
-import pytz
-from datetime import date, time, datetime, timedelta
-from django.conf import settings
-import imgurpython
-import PyRIGS.settings
-import sys
-import pathlib
-import inspect
+from envparse import env
 
 
 def create_datetime(year, month, day, hour, min):
@@ -19,15 +20,21 @@ def create_datetime(year, month, day, hour, min):
 
 
 def create_browser():
-    options = webdriver.ChromeOptions()
-    options.add_argument("--window-size=1920,1080")
-    # No caching, please and thank you
-    options.add_argument("--aggressive-cache-discard")
-    options.add_argument("--disk-cache-size=0")
-    options.add_argument("--headless")
-    if settings.CI:
-        options.add_argument("--no-sandbox")
-    driver = webdriver.Chrome(options=options)
+    browser = env('BROWSER', default="chrome")
+    if browser == "firefox":
+        options = webdriver.FirefoxOptions()
+        options.headless = True
+        driver = webdriver.Firefox(options=options)
+        driver.set_window_position(0, 0)
+        # Firefox is pissy about out of bounds otherwise
+        driver.set_window_size(3840, 2160)
+    else:
+        options = webdriver.ChromeOptions()
+        options.add_argument("--window-size=1920,1080")
+        options.add_argument("--headless")
+        if settings.CI:
+            options.add_argument("--no-sandbox")
+        driver = webdriver.Chrome(options=options)
     return driver
 
 
@@ -35,6 +42,7 @@ class BaseTest(LiveServerTestCase):
     def setUp(self):
         super().setUpClass()
         self.driver = create_browser()
+        self.wait = WebDriverWait(self.driver, 15)
 
     def tearDown(self):
         super().tearDown()
@@ -48,8 +56,8 @@ class AutoLoginTest(BaseTest):
             username="EventTest", first_name="Event", last_name="Test", initials="ETU", is_superuser=True)
         self.profile.set_password("EventTestPassword")
         self.profile.save()
-        loginPage = pages.LoginPage(self.driver, self.live_server_url).open()
-        loginPage.login("EventTest", "EventTestPassword")
+        login_page = pages.LoginPage(self.driver, self.live_server_url).open()
+        login_page.login("EventTest", "EventTestPassword")
 
 
 def screenshot_failure(func):
@@ -62,20 +70,9 @@ def screenshot_failure(func):
             if not pathlib.Path("screenshots").is_dir():
                 os.mkdir("screenshots")
             self.driver.save_screenshot(screenshot_file)
-
-            if settings.IMGUR_UPLOAD_CLIENT_ID != "":
-                config = {
-                    'album': None,
-                    'name': screenshot_name,
-                    'title': screenshot_name,
-                    'description': ""
-                }
-                client = imgurpython.ImgurClient(settings.IMGUR_UPLOAD_CLIENT_ID, settings.IMGUR_UPLOAD_CLIENT_SECRET)
-                image = client.upload_from_path(screenshot_file, config=config)
-                print("Error in test {} is at url {}".format(screenshot_name, image['link']), file=sys.stderr)
-            else:
-                print("Error in test {} is at path {}".format(screenshot_name, screenshot_file), file=sys.stderr)
+            print("Error in test {} is at path {}".format(screenshot_name, screenshot_file), file=sys.stderr)
             raise e
+
     return wrapper_func
 
 
@@ -86,12 +83,5 @@ def screenshot_failure_cls(cls):
     return cls
 
 
-# Checks if animation is done
-class animation_is_finished():
-    def __call__(self, driver):
-        numberAnimating = driver.execute_script('return $(":animated").length')
-        finished = numberAnimating == 0
-        if finished:
-            import time
-            time.sleep(0.1)
-        return finished
+def assert_times_equal(first_time, second_time):
+    assert first_time.replace(microsecond=0, second=0) == second_time.replace(microsecond=0, second=0)

PyRIGS/tests/pages.py

@@ -1,8 +1,8 @@
-from pypom import Page, Region
+from pypom import Page
+from selenium.common.exceptions import NoSuchElementException
 from selenium.webdriver.common.action_chains import ActionChains
 from selenium.webdriver.common.by import By
-from selenium.webdriver import Chrome
-from selenium.common.exceptions import NoSuchElementException
+
 from PyRIGS.tests import regions
 
 
@@ -44,6 +44,7 @@ class FormPage(BasePage):
         submit = self.find_element(*self._submit_locator)
         ActionChains(self.driver).move_to_element(submit).perform()
         submit.click()
+        self.wait.until(animation_is_finished())
         self.wait.until(lambda x: self.errors != previous_errors or self.success)
 
     @property
@@ -73,3 +74,13 @@ class LoginPage(BasePage):
         password_element.send_keys(password)
 
         self.find_element(*self._submit_locator).click()
+
+
+class animation_is_finished():
+    def __call__(self, driver):
+        number_animating = driver.execute_script('return $(":animated").length')
+        finished = number_animating == 0
+        if finished:
+            import time
+            time.sleep(0.1)
+        return finished

PyRIGS/tests/regions.py

@@ -1,15 +1,13 @@
-from pypom import Region
-from django.utils import timezone
-from django.conf import settings
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions
-from selenium.webdriver.remote.webelement import WebElement
-from selenium.webdriver.support.ui import WebDriverWait
-from selenium.webdriver.support.select import Select
-from selenium.webdriver.common.keys import Keys
-from selenium.common.exceptions import NoSuchElementException
 import datetime
 
+from django.conf import settings
+from pypom import Region
+from selenium.common.exceptions import NoSuchElementException
+from selenium.webdriver.common.by import By
+from selenium.webdriver.common.keys import Keys
+from selenium.webdriver.support import expected_conditions
+from selenium.webdriver.support.select import Select
+
 
 def parse_bool_from_string(string):
     # Used to convert from attribute strings to boolean values, written after I found this:
@@ -77,7 +75,6 @@ class BootstrapSelectElement(Region):
         self.open()
         search_box.clear()
         search_box.send_keys(query)
-        status_text = self.find_element(*self._status_locator)
         self.wait.until(expected_conditions.invisibility_of_element_located(self._status_locator))
 
     @property

PyRIGS/tests/test_unit.py

@@ -0,0 +1,65 @@
+from PyRIGS import urls
+from assets.tests.test_unit import create_asset_one
+import pytest
+from django.urls import URLPattern, URLResolver, reverse
+from django.urls.exceptions import NoReverseMatch
+from pytest_django.asserts import assertContains, assertRedirects, assertTemplateUsed, assertInHTML
+
+pytestmark = pytest.mark.django_db
+
+
+def find_urls_recursive(patterns):
+    urls_to_check = []
+    for url in patterns:
+        if isinstance(url, URLResolver):
+            urls_to_check += find_urls_recursive(url.url_patterns)
+        elif isinstance(url, URLPattern):
+            # Skip some thinks that actually don't need auth (mainly OEmbed JSONs that are essentially just a redirect)
+            if url.name is not None and url.name != "closemodal" and "json" not in str(url):
+                urls_to_check.append(url)
+    return urls_to_check
+
+
+def get_request_url(url):
+    pattern = str(url.pattern)
+    request_url = ""
+    try:
+        kwargz = {}
+        if ":pk>" in pattern:
+            kwargz['pk'] = 1
+        if ":model>" in pattern:
+            kwargz['model'] = "event"
+        return reverse(url.name, kwargs=kwargz)
+    except NoReverseMatch:
+        print("Couldn't test url " + pattern)
+
+
+def test_unauthenticated(client):  # Nothing should be available to the unauthenticated
+    create_asset_one()
+    for url in find_urls_recursive(urls.urlpatterns):
+        request_url = get_request_url(url)
+        if request_url and 'user' not in request_url:  # User module is full of edge cases
+            response = client.get(request_url, follow=True, HTTP_HOST='example.com')
+            assertContains(response, 'Login')
+            if 'application/json+oembed' in response.content.decode():
+                assertTemplateUsed(response, 'login_redirect.html')
+            else:
+                if "embed" in str(url):
+                    expected_url = "{0}?next={1}".format(reverse('login_embed'), request_url)
+                else:
+                    expected_url = "{0}?next={1}".format(reverse('login'), request_url)
+                assertRedirects(response, expected_url)
+
+
+def test_page_titles(admin_client):
+    create_asset_one()
+    for url in filter((lambda u: "embed" not in u.name), find_urls_recursive(urls.urlpatterns)):
+        request_url = get_request_url(url)
+        response = admin_client.get(request_url)
+        if hasattr(response, "context_data") and "page_title" in response.context_data:
+            expected_title = response.context_data["page_title"]
+            # try:
+            assertInHTML('<title>{} | Rig Information Gathering System'.format(expected_title), response.content.decode())
+            print("{} | {}".format(request_url, expected_title))  # If test fails, tell me where!
+            # except:
+            #    print(response.content.decode(), file=open('output.html', 'w'))

PyRIGS/urls.py

@@ -1,16 +1,11 @@
-from django.urls import path
-from django.conf.urls import include, url
-from django.contrib import admin
-from django.contrib.staticfiles.urls import staticfiles_urlpatterns
-from django.contrib.auth.decorators import login_required
 from django.conf import settings
-from django.views.decorators.clickjacking import xframe_options_exempt
-from django.contrib.auth.views import LoginView
+from django.conf.urls import include
+from django.contrib import admin
+from django.contrib.auth.decorators import login_required
+from django.contrib.staticfiles.urls import staticfiles_urlpatterns
+from django.urls import path
 from django.views.generic import TemplateView
-from PyRIGS.decorators import permission_required_with_403
-import RIGS
-import users
-import versioning
+
 from PyRIGS import views
 
 urlpatterns = [
@@ -27,7 +22,7 @@ urlpatterns = [
          name="api_secure"),
 
     path('closemodal/', views.CloseModal.as_view(), name='closemodal'),
-    path('search_help/', views.SearchHelp.as_view(), name='search_help'),
+    path('search_help/', login_required(views.SearchHelp.as_view()), name='search_help'),
 
     path('', include('users.urls')),
 
@@ -39,6 +34,6 @@ if settings.DEBUG:
 
     import debug_toolbar
     urlpatterns = [
-        url(r'^__debug__/', include(debug_toolbar.urls)),
+        path('__debug__/', include(debug_toolbar.urls)),
         path('bootstrap/', TemplateView.as_view(template_name="bootstrap.html")),
     ] + urlpatterns

PyRIGS/views.py

@@ -1,30 +1,27 @@
-from django.core.exceptions import PermissionDenied
-from django.http.response import HttpResponseRedirect
-from django.http import HttpResponse
-from django.urls import reverse_lazy, reverse, NoReverseMatch
-from django.views import generic
-from django.contrib.auth.views import LoginView
-from django.db.models import Q
-from django.shortcuts import get_object_or_404
-from django.core import serializers
-from django.conf import settings
-import simplejson
-from django.contrib import messages
 import datetime
-import pytz
 import operator
-from registration.views import RegistrationView
-from django.views.decorators.csrf import csrf_exempt
-
-from RIGS import models, forms
-from assets import models as asset_models
 from functools import reduce
 
-from django.views.decorators.cache import never_cache, cache_page
-from django.utils.decorators import method_decorator
+import simplejson
+from django.contrib import messages
+from django.core import serializers
+from django.core.exceptions import PermissionDenied
+from django.db.models import Q
+from django.http import HttpResponse
+from django.shortcuts import get_object_or_404
+from django.urls import reverse_lazy, reverse, NoReverseMatch
+from django.views import generic
 
+from RIGS import models
+from assets import models as asset_models
+
+
+def is_ajax(request):
+    return request.headers.get('x-requested-with') == 'XMLHttpRequest'
 
 # Displays the current rig count along with a few other bits and pieces
+
+
 class Index(generic.TemplateView):
     template_name = 'index.html'
 
@@ -151,7 +148,7 @@ class SecureAPIRequest(generic.View):
 
 class ModalURLMixin:
     def get_close_url(self, update, detail):
-        if self.request.is_ajax():
+        if is_ajax(self.request):
             url = reverse_lazy('closemodal')
             update_url = str(reverse_lazy(update, kwargs={'pk': self.object.pk}))
             messages.info(self.request, "modalobject=" + serializers.serialize("json", [self.object]))
@@ -170,7 +167,7 @@ class GenericListView(generic.ListView):
     def get_context_data(self, **kwargs):
         context = super(GenericListView, self).get_context_data(**kwargs)
         context['page_title'] = self.model.__name__ + "s"
-        if self.request.is_ajax():
+        if is_ajax(self.request):
             context['override'] = "base_ajax.html"
         return context
 
@@ -202,7 +199,7 @@ class GenericDetailView(generic.DetailView):
     def get_context_data(self, **kwargs):
         context = super(GenericDetailView, self).get_context_data(**kwargs)
         context['page_title'] = "{} | {}".format(self.model.__name__, self.object.name)
-        if self.request.is_ajax():
+        if is_ajax(self.request):
             context['override'] = "base_ajax.html"
         return context
 
@@ -213,7 +210,7 @@ class GenericUpdateView(generic.UpdateView):
     def get_context_data(self, **kwargs):
         context = super(GenericUpdateView, self).get_context_data(**kwargs)
         context['page_title'] = "Edit {}".format(self.model.__name__)
-        if self.request.is_ajax():
+        if is_ajax(self.request):
             context['override'] = "base_ajax.html"
         return context
 
@@ -224,7 +221,7 @@ class GenericCreateView(generic.CreateView):
     def get_context_data(self, **kwargs):
         context = super(GenericCreateView, self).get_context_data(**kwargs)
         context['page_title'] = "Create {}".format(self.model.__name__)
-        if self.request.is_ajax():
+        if is_ajax(self.request):
             context['override'] = "base_ajax.html"
         return context