Skip to content

GitLab

Commit f693c141 authored by dmorley's avatar dmorley
Browse files

status of all the things in the db

parent 94ade49b
Hide whitespace changes
Inline Side-by-side
Showing with 315 additions and 236 deletions
.gitlab-ci.yml
View file @ f693c141
......@@ -4,18 +4,17 @@ cache:
- vendor/
before_script:
- apt-get update -yqq
- apt-get install git -yqq
- apk add -U git
- docker-php-ext-install -j$(nproc) bcmath
- curl -sS https://getcomposer.org/installer | php
- php composer.phar install
test:7.2:
image: php:7.2-cli
image: php:7.2-cli-alpine
script:
- php composer.phar check-code
test:7.3-rc:
image: php:7.3-rc-cli
image: php:7.3-rc-cli-alpine
script:
- php composer.phar check-code
CHANGELOG.md
View file @ f693c141
......@@ -9,6 +9,21 @@ The format is based on [Keep a Changelog] and this project adheres to [Semantic
### Fixed
### Security
## [2.5.0]
:exclamation: DB migrations required! (see [SQL migration script][2.5.0-sql-migration])
### Added
- Meta data table for status and misc storage
- go.php can take a software= parameter and do a random pod on only one software/network
### Changed
- Status page is now /status from /db/status.php
### Deprecated
- db/pull.sh is no longer needed or used, cron can use db/update.php directly
### Removed
- db/last.data can be deleted from your system
### Fixed
- Moved status to a new table vs flat file as updates do not work right if not run every hour
### Security
## [2.4.2] - 2018-10-18
### Fixed
- Fixed db setup calls for cleaner links
......@@ -116,12 +131,14 @@ The format is based on [Keep a Changelog] and this project adheres to [Semantic
### Fixed
- Fix ipv6
[2.5.0-sql-migration]: https://git.feneas.org/diasporg/Poduptime/blob/master/db/migrations/2.4.1-2.5.0.sql
[2.4.1-sql-migration]: https://git.feneas.org/diasporg/Poduptime/blob/master/db/migrations/2.4.0-2.4.1.sql
[2.4.0-sql-migration]: https://git.feneas.org/diasporg/Poduptime/blob/master/db/migrations/2.3.0-2.4.0.sql
[2.3.0-sql-migration]: https://git.feneas.org/diasporg/Poduptime/blob/master/db/migrations/2.2.0-2.3.0.sql
[2.2.0-sql-migration]: https://git.feneas.org/diasporg/Poduptime/blob/master/db/migrations/2.1.4-2.2.0.sql
[Unreleased]: https://git.feneas.org/diasporg/Poduptime/compare/master...develop
[2.5.0]: https://git.feneas.org/diasporg/Poduptime/compare/2.4.1...2.5.0
[2.4.1]: https://git.feneas.org/diasporg/Poduptime/compare/2.4.0...2.4.1
[2.4.0]: https://git.feneas.org/diasporg/Poduptime/compare/2.3.1...2.4.0
[2.3.1]: https://git.feneas.org/diasporg/Poduptime/compare/2.3.0...2.3.1
......
README.md
View file @ f693c141
......@@ -44,15 +44,15 @@ psql -U podupuser podupdb < db/tables.sql
1. Edit `config.php` to add your DB and file settings
2. Touch add.log in location you configured in config.php
3. Create your backup folder
4. Add a pod and run `db/pull.sh init`
4. Add a pod and run `php db/update.php init`
# To Use:
run `db/pull.sh` manually or with cron to update your data
run `db/pull.sh debug` to debug output
run `db/pull.sh sqldebug` to debug sql
run `db/pull.sh develop` to run without email alerts to end users
run `db/pull.sh Check_System_Deleted` to re-check system deleted pods as needed
run `php db/update.php` to update your data
run `php db/update.php debug` to debug output
run `php db/update.php sqldebug` to debug sql
run `php db/update.php develop` to run without email alerts to end users
run `php db/update.php Check_System_Deleted` to re-check system deleted pods as needed
These commands can be combined
# To Upgrade:
......@@ -66,7 +66,8 @@ psql -U podupuser podupdb < db/migrations/xxx.sql (see db/migrations/README.md f
# Status
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/e4fdde7b4ec34c56a86938d3ba6e3983)](https://www.codacy.com/app/diasporg/Poduptime?utm_source=git.feneas.org&amp;utm_medium=referral&amp;utm_content=diasporg/Poduptime&amp;utm_campaign=Badge_Grade)
[![Build status](https://badge.buildkite.com/2a2d6f9ff932fc359416fff31ee72c3dce703569a801f83af6.svg)](https://buildkite.com/diasporg/poduptime)
[![pipeline status](https://git.feneas.org/diasporg/Poduptime/badges/develop/pipeline.svg)](https://git.feneas.org/diasporg/Poduptime/commits/develop)
[![coverage report](https://git.feneas.org/diasporg/Poduptime/badges/develop/coverage.svg)](https://git.feneas.org/diasporg/Poduptime/commits/develop)
============================
......
composer.json
View file @ f693c141
......@@ -35,7 +35,7 @@
},
"scripts": {
"check-code": [
"vendor/bin/phpcs --standard=PSR2 -snp --encoding=utf-8 --report-width=150 lib *.php"
"vendor/bin/phpcs --standard=PSR2 -snp --encoding=utf-8 --report-width=150 db lib *.php"
]
}
}
composer.lock
View file @ f693c141
{
"_readme": [
"This file locks the dependencies of your project to a known state",
"Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies",
"Read more about it at https://getcomposer.org/doc/01-basic-usage.md#composer-lock-the-lock-file",
"This file is @generated automatically"
],
"content-hash": "1ce82c5590f5a4ce2bf5a2b8812da707",
......@@ -149,16 +149,16 @@
},
{
"name": "gabordemooij/redbean",
"version": "v5.1",
"version": "v5.2",
"source": {
"type": "git",
"url": "https://github.com/gabordemooij/redbean.git",
"reference": "a02e58ad5519149f572559a8293b5dffea385956"
"reference": "4ec3ee400bfbe4c52da4a376de9a6c338b5a8819"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/gabordemooij/redbean/zipball/a02e58ad5519149f572559a8293b5dffea385956",
"reference": "a02e58ad5519149f572559a8293b5dffea385956",
"url": "https://api.github.com/repos/gabordemooij/redbean/zipball/4ec3ee400bfbe4c52da4a376de9a6c338b5a8819",
"reference": "4ec3ee400bfbe4c52da4a376de9a6c338b5a8819",
"shasum": ""
},
"require": {
......@@ -172,7 +172,7 @@
},
"notification-url": "https://packagist.org/downloads/",
"license": [
"New BSD and GPLv2"
"BSD-3-Clause"
],
"authors": [
{
......@@ -186,7 +186,7 @@
"keywords": [
"orm"
],
"time": "2018-04-01T11:51:37+00:00"
"time": "2018-10-26T09:06:03+00:00"
},
{
"name": "geoip2/geoip2",
......@@ -242,16 +242,16 @@
},
{
"name": "jaybizzle/crawler-detect",
"version": "v1.2.71",
"version": "v1.2.74",
"source": {
"type": "git",
"url": "https://github.com/JayBizzle/Crawler-Detect.git",
"reference": "bfb7d00c70d0ad59037bed7f08a2b6c10b1ed2ba"
"reference": "e5fbc685383a4b8f49183b3a646b193491cc14c5"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/JayBizzle/Crawler-Detect/zipball/bfb7d00c70d0ad59037bed7f08a2b6c10b1ed2ba",
"reference": "bfb7d00c70d0ad59037bed7f08a2b6c10b1ed2ba",
"url": "https://api.github.com/repos/JayBizzle/Crawler-Detect/zipball/e5fbc685383a4b8f49183b3a646b193491cc14c5",
"reference": "e5fbc685383a4b8f49183b3a646b193491cc14c5",
"shasum": ""
},
"require": {
......@@ -287,7 +287,7 @@
"crawlerdetect",
"php crawler detect"
],
"time": "2018-10-16T18:51:19+00:00"
"time": "2018-11-21T20:56:45+00:00"
},
{
"name": "longman/ip-tools",
......@@ -391,16 +391,16 @@
},
{
"name": "maxmind-db/reader",
"version": "v1.3.0",
"version": "v1.4.0",
"source": {
"type": "git",
"url": "https://github.com/maxmind/MaxMind-DB-Reader-php.git",
"reference": "e042b4f8a2dff41e19019faf16427178b07fbd58"
"reference": "2f1086e38939e808cc23b17daa469da094af9125"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/maxmind/MaxMind-DB-Reader-php/zipball/e042b4f8a2dff41e19019faf16427178b07fbd58",
"reference": "e042b4f8a2dff41e19019faf16427178b07fbd58",
"url": "https://api.github.com/repos/maxmind/MaxMind-DB-Reader-php/zipball/2f1086e38939e808cc23b17daa469da094af9125",
"reference": "2f1086e38939e808cc23b17daa469da094af9125",
"shasum": ""
},
"require": {
......@@ -443,7 +443,7 @@
"geolocation",
"maxmind"
],
"time": "2018-02-21T21:23:33+00:00"
"time": "2018-11-20T16:04:10+00:00"
},
{
"name": "maxmind/web-service-common",
......@@ -493,16 +493,16 @@
},
{
"name": "nesbot/carbon",
"version": "1.34.0",
"version": "1.36.1",
"source": {
"type": "git",
"url": "https://github.com/briannesbitt/Carbon.git",
"reference": "1dbd3cb01c5645f3e7deda7aa46ef780d95fcc33"
"reference": "63da8cdf89d7a5efe43aabc794365f6e7b7b8983"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/briannesbitt/Carbon/zipball/1dbd3cb01c5645f3e7deda7aa46ef780d95fcc33",
"reference": "1dbd3cb01c5645f3e7deda7aa46ef780d95fcc33",
"url": "https://api.github.com/repos/briannesbitt/Carbon/zipball/63da8cdf89d7a5efe43aabc794365f6e7b7b8983",
"reference": "63da8cdf89d7a5efe43aabc794365f6e7b7b8983",
"shasum": ""
},
"require": {
......@@ -510,9 +510,12 @@
"symfony/translation": "~2.6 || ~3.0 || ~4.0"
},
"require-dev": {
"friendsofphp/php-cs-fixer": "~2",
"phpunit/phpunit": "^4.8.35 || ^5.7"
},
"suggest": {
"friendsofphp/php-cs-fixer": "Needed for the `composer phpcs` command. Allow to automatically fix code style.",
"phpstan/phpstan": "Needed for the `composer phpstan` command. Allow to detect potential errors."
},
"type": "library",
"extra": {
"laravel": {
......@@ -544,7 +547,7 @@
"datetime",
"time"
],
"time": "2018-09-20T19:36:25+00:00"
"time": "2018-11-22T18:23:02+00:00"
},
{
"name": "noplanman/xec",
......@@ -676,38 +679,46 @@
"time": "2018-10-05T07:34:25+00:00"
},
{
"name": "rinvex/country",
"version": "v3.1.0",
"name": "symfony/contracts",
"version": "v1.0.1",
"source": {
"type": "git",
"url": "https://github.com/rinvex/country.git",
"reference": "e32228ef43f26d3b02296be9454f842c52d492f3"
"url": "https://github.com/symfony/contracts.git",
"reference": "3edf0ab943d1985a356721952cba36ff31bd6e5f"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/rinvex/country/zipball/e32228ef43f26d3b02296be9454f842c52d492f3",
"reference": "e32228ef43f26d3b02296be9454f842c52d492f3",
"url": "https://api.github.com/repos/symfony/contracts/zipball/3edf0ab943d1985a356721952cba36ff31bd6e5f",
"reference": "3edf0ab943d1985a356721952cba36ff31bd6e5f",
"shasum": ""
},
"require": {
"php": "^7.0.0"
"php": "^7.1.3"
},
"require-dev": {
"phpunit/phpunit": "^5.4.0"
"psr/cache": "^1.0",
"psr/container": "^1.0"
},
"suggest": {
"psr/cache": "When using the Cache contracts",
"psr/container": "When using the Service contracts",
"symfony/cache-contracts-implementation": "",
"symfony/service-contracts-implementation": "",
"symfony/translation-contracts-implementation": ""
},
"type": "library",
"extra": {
"branch-alias": {
"dev-master": "3.0-dev"
"dev-master": "1.0-dev"
}
},
"autoload": {
"files": [
"src/helpers.php"
],
"psr-4": {
"Rinvex\\Country\\": "src/"
}
"Symfony\\Contracts\\": ""
},
"exclude-from-classmap": [
"**/Tests/"
]
},
"notification-url": "https://packagist.org/downloads/",
"license": [
......@@ -715,52 +726,38 @@
],
"authors": [
{
"name": "Rinvex LLC",
"email": "help@rinvex.com",
"homepage": "https://rinvex.com"
},
{
"name": "Abdelrahman Omran",
"email": "me@omranic.com",
"homepage": "https://omranic.com",
"role": "Project Lead"
"name": "Nicolas Grekas",
"email": "p@tchwork.com"
},
{
"name": "The Generous Laravel Community",
"homepage": "https://github.com/rinvex/country/contributors"
"name": "Symfony Community",
"homepage": "https://symfony.com/contributors"
}
],
"description": "Rinvex Country is a simple and lightweight package for retrieving country details with flexibility. A whole bunch of data including name, demonym, capital, iso codes, dialling codes, geo data, currencies, flags, emoji, and other attributes for all 250 countries worldwide at your fingertips.",
"homepage": "https://rinvex.com",
"description": "A set of abstractions extracted out of the Symfony components",
"homepage": "https://symfony.com",
"keywords": [
"Flexible",
"Simple",
"countries",
"country",
"currencies",
"demonym",
"dialling",
"emoji",
"flags",
"geographic",
"languages",
"rinvex",
"svg"
"abstractions",
"contracts",
"decoupling",
"interfaces",
"interoperability",
"standards"
],
"time": "2017-03-07T18:40:20+00:00"
"time": "2018-11-24T09:35:08+00:00"
},
{
"name": "symfony/polyfill-mbstring",
"version": "v1.9.0",
"version": "v1.10.0",
"source": {
"type": "git",
"url": "https://github.com/symfony/polyfill-mbstring.git",
"reference": "d0cd638f4634c16d8df4508e847f14e9e43168b8"
"reference": "c79c051f5b3a46be09205c73b80b346e4153e494"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/symfony/polyfill-mbstring/zipball/d0cd638f4634c16d8df4508e847f14e9e43168b8",
"reference": "d0cd638f4634c16d8df4508e847f14e9e43168b8",
"url": "https://api.github.com/repos/symfony/polyfill-mbstring/zipball/c79c051f5b3a46be09205c73b80b346e4153e494",
"reference": "c79c051f5b3a46be09205c73b80b346e4153e494",
"shasum": ""
},
"require": {
......@@ -806,24 +803,25 @@
"portable",
"shim"
],
"time": "2018-08-06T14:22:27+00:00"
"time": "2018-09-21T13:07:52+00:00"
},
{
"name": "symfony/translation",
"version": "v4.1.6",
"version": "v4.2.0",
"source": {
"type": "git",
"url": "https://github.com/symfony/translation.git",
"reference": "9f0b61e339160a466ebcde167a6c5521c810e304"
"reference": "ff9a878c9b8f8bcd4d9138e2d32f508c942773d9"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/symfony/translation/zipball/9f0b61e339160a466ebcde167a6c5521c810e304",
"reference": "9f0b61e339160a466ebcde167a6c5521c810e304",
"url": "https://api.github.com/repos/symfony/translation/zipball/ff9a878c9b8f8bcd4d9138e2d32f508c942773d9",
"reference": "ff9a878c9b8f8bcd4d9138e2d32f508c942773d9",
"shasum": ""
},
"require": {
"php": "^7.1.3",
"symfony/contracts": "^1.0",
"symfony/polyfill-mbstring": "~1.0"
},
"conflict": {
......@@ -831,6 +829,9 @@
"symfony/dependency-injection": "<3.4",
"symfony/yaml": "<3.4"
},
"provide": {
"symfony/translation-contracts-implementation": "1.0"
},
"require-dev": {
"psr/log": "~1.0",
"symfony/config": "~3.4|~4.0",
......@@ -848,7 +849,7 @@
"type": "library",
"extra": {
"branch-alias": {
"dev-master": "4.1-dev"
"dev-master": "4.2-dev"
}
},
"autoload": {
......@@ -875,7 +876,7 @@
],
"description": "Symfony Translation Component",
"homepage": "https://symfony.com",
"time": "2018-10-02T16:36:10+00:00"
"time": "2018-11-27T07:20:32+00:00"
}
],
"packages-dev": [
......
config.php.example
View file @ f693c141
......@@ -35,13 +35,13 @@ return [
//DNS server for dnssec testing. 1.1.1.1 tests the best
'dnsserver' => '',
//CA for curl to use - full file path (pull.sh will update this monthly)
//CA for curl to use - full file path
'cafullpath' => '',
//Mapbox.com API key. https://www.mapbox.com/help/how-access-tokens-work/
'mapboxkey' => '',
//Geolite2-city database file in mmdb format - full file path (pull.sh will update this monthly)
//Geolite2-city database file in mmdb format - full file path
'geoip2db' => '',
//detectlanguage.com api key
......
db/add.php
View file @ f693c141
......@@ -29,7 +29,7 @@ try {
FROM pods
');
} catch (\RedBeanPHP\RedException $e) {
die('Error in SQL query: ' . $e->getMessage());
debug('Error in SQL query: ' . sprintf($e->getMessage()));
}
$stop = false;
......@@ -55,7 +55,7 @@ foreach ($pods as $pod) {
R::store($p);
} catch (\RedBeanPHP\RedException $e) {
die('Error in SQL query: ' . $e->getMessage());
debug('Error in SQL query: ' . sprintf($e->getMessage()));
}
echo <<<EOF
......@@ -88,17 +88,9 @@ if (!$stop) {
$link = max($info['links'])['href'];
}
$chss = curl_init();
curl_setopt($chss, CURLOPT_URL, $link);
curl_setopt($chss, CURLOPT_POST, 0);
curl_setopt($chss, CURLOPT_HEADER, 0);
curl_setopt($chss, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($chss, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($chss, CURLOPT_NOBODY, 0);
$outputssl = curl_exec($chss);
curl_close($chss);
if ($outputssl && stripos($outputssl, 'openRegistrations') !== false) {
$output = curl($link, true);
if ($output['body'] && stripos($output['body'], 'openRegistrations') !== false) {
echo 'Your pod has ssl and is valid<br>';
$publickey = md5(uniqid($_domain, true));
......@@ -114,7 +106,7 @@ if (!$stop) {
R::store($p);
} catch (\RedBeanPHP\RedException $e) {
die('Error in SQL query: ' . $e->getMessage());
debug('Error in SQL query: ' . sprintf($e->getMessage()));
}
if ($_email) {
......@@ -132,7 +124,6 @@ if (!$stop) {
}
echo 'Data successfully inserted! Your pod will be checked and live on the list in a few hours!';
} else {
echo 'Could not validate your pod, check your setup!<br>Take a look at <a href="' . $link . '">your /nodeinfo</a>';
}
......
db/backup.php
View file @ f693c141
......@@ -16,7 +16,7 @@ if (!is_cli()) {
$keep_for = 60 * 60 * 6; // 6 hours
$backup_file = c('backup_dir') . '/dump_' . date('Ymd_His') . '.sql';
printf("Making backup of '%s' to '%s'...", c('pgdb'), $backup_file);
debug("Making backup of '%s' to '%s'...", c('pgdb'), $backup_file);
system(sprintf(
'export PGPASSWORD=%3$s &&' .
'"%1$s" --clean --format=tar --username=%2$s %4$s >> "%5$s"',
......@@ -26,7 +26,7 @@ system(sprintf(
c('pgdb'),
$backup_file
), $exit_code);
printf(" %s\n", $exit_code === 0 ? 'Success!' : 'Failed.');
debug(" %s\n", $exit_code === 0 ? 'Success!' : 'Failed.');
$dirh = dir(c('backup_dir'));
while ($entry = $dirh->read()) {
......@@ -48,3 +48,5 @@ while ($entry = $dirh->read()) {
unlink($file) ? 'Success!' : 'Failed.'
);
}
updateMeta('backup');
db/migrations/2.4.1-2.5.0.sql 0 → 100644
View file @ f693c141
CREATE TABLE meta (
id serial8 UNIQUE PRIMARY KEY,
name text,
value text,
date_created timestamp DEFAULT current_timestamp
);
INSERT INTO meta (name) VALUES('languages_updated');
INSERT INTO meta (name) VALUES('cacert_updated');
INSERT INTO meta (name) VALUES('geoip_updated');
INSERT INTO meta (name) VALUES('masterversions_updated');
INSERT INTO meta (name) VALUES('federation_updated');
INSERT INTO meta (name) VALUES('statstable_updated');
INSERT INTO meta (name) VALUES('backup');
INSERT INTO meta (name) VALUES('pods_updated');
db/pull.sh deleted 100755 → 0
View file @ 94ade49b
#!/usr/bin/env bash
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
cd "$SCRIPT_DIR"
FLAG_FILE="/tmp/poduptime.updating"
HOUR=$(date +%H)
DAY=$(date +%d)
HAPPY=" :-)"
SAD=" :-("
if [ -f "$FLAG_FILE" ]; then
echo "Already running update"
exit 1
fi
touch "$FLAG_FILE"
# https://stackoverflow.com/a/26820300
printf "%s" "Checking for internet..."
if ! wget -q --spider --tries=2 --timeout=15 https://www.google.com; then
echo "$SAD"
echo "Could not update pods as no internet"
rm "$FLAG_FILE"
exit 1
fi
echo "$HAPPY"
if [ "$HOUR" = 23 ] || [ "$1" = 'init' ]; then
printf "%s" "Updating master versions..."
if php update-masterversions.php; then
echo "$HAPPY"
else
echo "$SAD"
fi
printf "%s" "Updating Monthy Stats Table..."
if php update-monthly-stats.php; then
echo "$HAPPY"
else
echo "$SAD"
fi
printf "%s" "Crawling and updating from the federation..."
if php update-remote-data.php; then
echo "$HAPPY"
else
echo "$SAD"
fi
if [ "$DAY" = 23 ] || [ "$1" = 'init' ]; then
printf "%s" "Updating CA..."
CACERT_FILE="$(php -r "echo (require __DIR__ . '/../config.php')['cafullpath'];")"
if curl -Lss https://curl.haxx.se/ca/cacert.pem -o "$CACERT_FILE"; then
echo "$HAPPY"
else
echo "$SAD"
fi
printf "%s" "Updating GeoIP2 DB..."
GEODB_FILE="$(php -r "echo (require __DIR__ . '/../config.php')['geoip2db'];")"
if funzip <(curl -Lss http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz) > "$GEODB_FILE"; then
echo "$HAPPY"
else
echo "$SAD"
fi
fi
fi
echo "Updating in new pod data...";
php update.php $@
touch last.data
echo "Finished update!"
echo "Backing up..."
php backup.php
echo
rm "$FLAG_FILE"
db/release-update-lock.php 0 → 100644
View file @ f693c141
<?php
declare(strict_types=1);
require_once __DIR__ . '/../boot.php';
updateMeta('pods_updating', false);
db/status.php deleted 100644 → 0
View file @ 94ade49b
<?php
/**
* Check status of previously pulled data.
*/
declare(strict_types=1);
$dur = time() - filemtime('last.data');
echo $dur;
if ($dur > 4500) {
http_response_code(500);
} else {
http_response_code(200);