1
0
mirror of https://github.com/snobu/destreamer.git synced 2026-01-29 11:22:18 +00:00

20 Commits

Author SHA1 Message Date
lukaarma
c81b16c8f3 Merge pull request #322 from apandada1/patch-1
update README - add instructions for Linux
2021-02-09 21:33:34 +01:00
Archisman Panigrahi
aa569bee4d update README - add instructions for Linux
These instructions are valid for a lot of Linux distros, and not just limited to Raspberry Pi.
2021-02-07 12:20:18 +05:30
Adrian Calinescu
9ebd4faab3 Add Raspberry Pi note 2021-01-26 16:59:48 +02:00
dependabot[bot]
66c018e164 Bump axios from 0.21.0 to 0.21.1 (#309)
Bumps [axios](https://github.com/axios/axios) from 0.21.0 to 0.21.1.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v0.21.1/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v0.21.0...v0.21.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-01-26 14:52:39 +02:00
Adrian Calinescu
3c7d61febe Fix return HTTP 403 reason with or without verbose (#315) 2021-01-26 14:49:42 +02:00
Adrian Calinescu
b5226713b6 If you're using Node 8 you're on your own, pal. 2021-01-13 20:55:21 +02:00
snobu
f9bc0c7128 I've had enough of Node 8.x horse manure 2021-01-13 20:53:00 +02:00
snobu
2c38517bcd Fix some dumb npm s*** 2021-01-13 20:18:50 +02:00
lukaarma
f8207f4fd1 Group parsing fix and error out on old ffmpeg version (#298)
* fixed parsing for group with more than 100 videos

* updated all packages to latest version

* Error on old ffmpeg binaries (closes #294)
minor linting fixes

* automatic update of files

Co-authored-by: Adrian Calinescu <foo@snobu.org>
2021-01-13 20:12:12 +02:00
lukaarma
58122d5c4e Merge pull request #307 from snobu/dependabot/npm_and_yarn/axios-0.21.1
Bump axios from 0.19.2 to 0.21.1 (vulnerability fix)
2021-01-08 16:16:23 +01:00
Adrian Calinescu
0726bd90f4 Holiday cleanup 2021-01-07 18:33:57 +02:00
dependabot[bot]
2d0407e5c8 Bump axios from 0.19.2 to 0.21.1
Bumps [axios](https://github.com/axios/axios) from 0.19.2 to 0.21.1.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v0.21.1/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v0.19.2...v0.21.1)

Signed-off-by: dependabot[bot] <support@github.com>
2021-01-05 23:14:38 +00:00
Adrian Calinescu
0da9c6fb5f Fixup for the holidays 2020-12-22 21:51:07 +02:00
Adrian Calinescu
f26204c38a Revert "Fixed parsing for group with more than 100 videos (#288)" (#295)
This reverts commit cd1ac82fea.
2020-12-15 13:02:23 +02:00
lukaarma
cd1ac82fea Fixed parsing for group with more than 100 videos (#288)
* fixed parsing for group with more than 100 videos
* updated all packages to latest version
2020-12-15 12:55:10 +02:00
Adrian Calinescu
fbe8de00de Added link to aria branch in README 2020-12-03 00:07:13 +02:00
fulminemizzega
b48af65285 fix quotes in input file example in README.md (#283)
This should close #281
2020-11-22 20:54:42 +02:00
snobu
e9070511cf Srsly really fixed group download this time 2020-11-14 20:08:45 +02:00
snobu
ad483f3eb7 Fix group download, now limited to first 100 videos 2020-11-14 19:54:50 +02:00
Adrian Calinescu
ac0fdf5468 Fix MSEdge launch params 2020-10-15 16:07:20 +03:00
19 changed files with 783 additions and 5834 deletions

View File

@@ -14,7 +14,7 @@ jobs:
strategy:
matrix:
node-version: [8.x, 10.x, 12.x, 13.x]
node-version: [10.x, 12.x, 13.x]
steps:
- uses: actions/checkout@v1

2
.gitignore vendored
View File

@@ -4,8 +4,6 @@
*.js
*.zip
.vscode\launch.json
.chrome_data
node_modules
videos

20
.vscode/launch.json vendored Normal file
View File

@@ -0,0 +1,20 @@
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"type": "node",
"runtimeArgs": ["--max-http-header-size", "32768"],
"request": "launch",
"name": "Launch Program",
"program": "${workspaceFolder}/build/src/destreamer.js",
"args": [
"-i",
"https://web.microsoftstream.com/video/ce4da1ff-0400-86ec-2ad6-f1ea83412074" // Bacon and Eggs
]
}
]
}

View File

@@ -1,3 +1,4 @@
{
"eslint.enable": true
}
"eslint.enable": true,
"typescript.tsdk": "node_modules\\typescript\\lib"
}

View File

@@ -2,9 +2,7 @@
<img src="https://github.com/snobu/destreamer/workflows/Node%20CI/badge.svg" alt="CI build status" />
</a>
# destreamer v3.0 (aria2c as download manager)
## This is a pre-release branch so don't expect stability, do expect speed improvements. Tons of it.
**destreamer v3.0** is just around the corner. You can try out a pre-release today by cloning [this branch](https://github.com/snobu/destreamer/tree/aria2c_forRealNow).
![destreamer](assets/logo.png)
@@ -40,12 +38,10 @@ Hopefully this doesn't break the end user agreement for Microsoft Stream. Since
## Prereqs
- [**Node.js**][node]: You'll need Node.js version 8.0 or higher. A GitHub Action runs tests on all major Node versions on every commit. One caveat for Node 8, if you get a `Parse Error` with `code: HPE_HEADER_OVERFLOW` you're out of luck and you'll need to upgrade to Node 10+.
- [**Node.js**][node]: You'll need Node.js version 8.0 or higher. A GitHub Action runs tests on all major Node versions on every commit. One caveat for Node 8, if you get a `Parse Error` with `code: HPE_HEADER_OVERFLOW` you're out of luck and you'll need to upgrade to Node 10+. PLEASE NOTE WE NO LONGER TEST BUILDS AGAINST NODE 8.x. YOU ARE ON YOUR OWN.
- **npm**: usually comes with Node.js, type `npm` in your terminal to check for its presence
- [**ffmpeg**][ffmpeg]: a recent version (year 2019 or above), in `$PATH` or in the same directory as this README file (project root).
- [**git**][git]: one or more npm dependencies require git.
- [**aria2**][aria2]: present in your `$PATH`, on Linux you can install via `sudo apt install aria2`.
Destreamer takes a [honeybadger](https://www.youtube.com/watch?v=4r7wHMg5Yjg) approach towards the OS it's running on. We've successfully tested it on Windows, macOS and Linux.
@@ -78,10 +74,15 @@ const browser: puppeteer.Browser = await puppeteer.launch({
Now, change `executablePath` to reflect the path to your browser and profile (i.e. to use Microsoft Edge on Windows):
```typescript
executablePath: "'C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe' --profile-directory=Default",
executablePath: 'C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe',
```
In Linux for Chromium,
```typescript
executablePath: '/usr/bin/chromium-browser',
```
Depending on your distro, it may also be `/usr/bin/chromium`. You will have to change it appropriately for Google Chrome.
Note that for Mac/Linux the path will look a little different but no other changes are necessary.
Note that for Mac the path may look a little different but no other changes are necessary.
You need to rebuild (`npm run build`) every time you change this configuration.
@@ -230,7 +231,6 @@ Please open an [issue](https://github.com/snobu/destreamer/issues) and we'll loo
[xming]: https://sourceforge.net/projects/xming/
[node]: https://nodejs.org/en/download/
[git]: https://git-scm.com/downloads
[aria2]: https://aria2.github.io
[wsl]: https://github.com/snobu/destreamer/issues/90#issuecomment-619377950
[polimi]: https://www.polimi.it
[unipi]: https://www.unipi.it/

5573
package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@@ -7,9 +7,7 @@
"version": "2.1.0",
"description": "Save Microsoft Stream videos for offline enjoyment.",
"main": "build/src/destreamer.js",
"bin": {
"destreamer": "build/src/destreamer.js"
},
"bin": "build/src/destreamer.js",
"scripts": {
"build": "echo Transpiling TypeScript to JavaScript... && node node_modules/typescript/bin/tsc && echo Destreamer was built successfully.",
"test": "mocha build/test",
@@ -19,37 +17,34 @@
"author": "snobu",
"license": "MIT",
"devDependencies": {
"@types/cli-progress": "^3.8.0",
"@types/jwt-decode": "^2.2.1",
"@types/mocha": "^8.0.4",
"@types/puppeteer": "^5.4.0",
"@types/readline-sync": "^1.4.3",
"@types/tmp": "^0.2.0",
"@types/ws": "^7.4.0",
"@types/yargs": "^15.0.11",
"@typescript-eslint/eslint-plugin": "^4.9.0",
"@typescript-eslint/parser": "^4.9.0",
"eslint": "^7.14.0",
"mocha": "^8.2.1",
"typescript": "^4.1.2"
"tmp": "^0.2.1"
},
"dependencies": {
"axios": "^0.21.0",
"@tedconf/fessonia": "^2.1.2",
"@types/cli-progress": "^3.8.0",
"@types/jwt-decode": "^2.2.1",
"axios": "^0.21.1",
"axios-retry": "^3.1.9",
"cli-progress": "^3.8.2",
"colors": "^1.4.0",
"is-elevated": "^3.0.0",
"iso8601-duration": "^1.3.0",
"jwt-decode": "^3.1.2",
"m3u8-parser": "^4.5.0",
"portfinder": "^1.0.28",
"puppeteer": "5.5.0",
"readline-sync": "^1.4.10",
"sanitize-filename": "^1.6.3",
"terminal-image": "^1.2.1",
"tmp": "^0.2.1",
"typescript": "^4.1.2",
"winston": "^3.3.3",
"ws": "^7.4.0",
"yargs": "^16.1.1"
}
}

View File

@@ -15,7 +15,7 @@ export class ApiClient {
this.axiosInstance = axios.create({
baseURL: session?.ApiGatewayUri,
// timeout: 7000,
headers: { 'User-Agent': 'destreamer/3.0 (Preview)' }
headers: { 'User-Agent': 'destreamer/2.0 (Hammer of Dawn)' }
});
axiosRetry(this.axiosInstance, {
@@ -34,10 +34,9 @@ export class ApiClient {
return true;
}
logger.warn(`Got HTTP code ${err?.response?.status ?? undefined}. Retrying request...`);
logger.warn('Here is the error message: \n' +
JSON.stringify(err.response?.data ?? undefined) +
'\nRetrying request...');
logger.warn(`We called this URL: ${err.response?.config.baseURL}${err.response?.config.url}`);
logger.warn('Here is the error message: ');
console.dir(err.response?.data);
logger.warn('We called this URL: ' + err.response?.config.baseURL + err.response?.config.url);
const shouldRetry: boolean = retryCodes.includes(err?.response?.status ?? 0);
@@ -84,13 +83,6 @@ export class ApiClient {
'Authorization': 'Bearer ' + this.session?.AccessToken
};
logger.debug(
'[ApiClient.callApi]\n' +
'path: ' + path + '\n' +
'method: ' + method + '\n' +
'payload: ' + payload + '\n'
);
return this.axiosInstance?.request({
method: method,
headers: headers,
@@ -112,14 +104,6 @@ export class ApiClient {
'Authorization': 'Bearer ' + this.session?.AccessToken
};
logger.debug(
'[ApiClient.callUrl]\n' +
'url: ' + url + '\n' +
'method: ' + method + '\n' +
'payload: ' + payload + '\n' +
'responseType: ' + responseType + '\n'
);
return this.axiosInstance?.request({
method: method,
headers: headers,

View File

@@ -71,27 +71,32 @@ export const argv: any = yargs.options({
default: false,
demandOption: false
},
debug: {
alias: 'd',
describe: 'Set logging level to debug (only use this if you know what are doing)',
type: 'boolean',
default: false,
demandOption: false
},
selectQuality: {
alias: 'q',
describe: 'Select the quality with a number 1 (worst) trough 10 (best), 0 prompt the user for each video',
default: 10,
type: 'number',
demandOption: false
},
closedCaptions: {
alias: 'cc',
describe: 'Check if closed captions are available and let the user choose which one to download (will not ask if only one available)',
describe: 'Check if closed captions are available and let the user choose which one to download (will not ask if only one available).',
type: 'boolean',
default: false,
demandOption: false
},
noCleanup: {
alias: 'nc',
describe: 'Do not delete the downloaded video file when an FFmpeg error occurs.',
type: 'boolean',
default: false,
demandOption: false
},
vcodec: {
describe: 'Re-encode video track. Specify FFmpeg codec (e.g. libx265) or set to "none" to disable video.',
type: 'string',
default: 'copy',
demandOption: false
},
acodec: {
describe: 'Re-encode audio track. Specify FFmpeg codec (e.g. libopus) or set to "none" to disable audio.',
type: 'string',
default: 'copy',
demandOption: false
},
format: {
describe: 'Output container format (mkv, mp4, mov, anything that FFmpeg supports).',
type: 'string',
@@ -108,9 +113,17 @@ export const argv: any = yargs.options({
.wrap(120)
.check(() => noArguments())
.check((argv: any) => checkInputConflicts(argv.videoUrls, argv.inputFile))
.check((argv: any) => checkOutputDirectoryExistance(argv.outputDirectory))
.check((argv: any) => {
if (checkOutDir(argv.outputDirectory)) {
return true;
}
else {
logger.error(CLI_ERROR.INVALID_OUTDIR);
throw new Error(' ');
}
})
.check((argv: any) => isOutputTemplateValid(argv))
.check((argv: any) => checkQualityValue(argv))
.argv;
@@ -160,18 +173,6 @@ function checkInputConflicts(videoUrls: Array<string | number> | undefined,
}
function checkOutputDirectoryExistance(dir: string): boolean {
if (checkOutDir(dir)) {
return true;
}
else {
logger.error(CLI_ERROR.INVALID_OUTDIR, { fatal: true });
throw new Error(' ');
}
}
function isOutputTemplateValid(argv: any): boolean {
let finalTemplate: string = argv.outputTemplate;
const elementRegEx = RegExp(/{(.*?)}/g);
@@ -205,25 +206,6 @@ function isOutputTemplateValid(argv: any): boolean {
}
function checkQualityValue(argv: any): boolean {
if (isNaN(argv.selectQuality)) {
logger.error('The quality value provided was not a number, switching to default');
argv.selectQuality = 10;
return true;
}
else if (argv.selectQuality < 0 || argv.selectQuality > 10) {
logger.error('The quality value provided was outside the valid range, switching to default');
argv.selectQuality = 10;
return true;
}
else {
return true;
}
}
export function promptUser(choices: Array<string>): number {
const index: number = readlineSync.keyInSelect(choices, 'Which resolution/format do you prefer?');

View File

@@ -1,36 +0,0 @@
import { ApiClient } from './ApiClient';
import { logger } from './Logger';
import { Session } from './Types';
import crypto from 'crypto';
export async function getDecrypter(playlistUrl: string, session: Session): Promise<crypto.Decipher> {
const apiClient = ApiClient.getInstance(session);
const keyOption = await apiClient.callUrl(playlistUrl, 'get', null, 'text')
.then(res => (res?.data as string).split(/\r?\n/)
.find(line => line.startsWith('#EXT-X-KEY')));
if (keyOption) {
logger.debug('[Decrypter] CRIPTO LINE IN M3U8: ' + keyOption);
const match = RegExp(/#EXT-X-KEY:METHOD=(.*?),URI="(.*?)",IV=0X(.*)/).exec(keyOption);
if (!match) {
throw new Error('No match for regex');
}
const algorithm = match[1].toLowerCase().replace('-', '');
const key: Buffer = await apiClient.callUrl(match[2], 'post', null, 'arraybuffer')
.then(res => res?.data);
const iv = Buffer.from(match[3], 'hex');
return crypto.createDecipheriv(algorithm, key, iv);
}
else {
process.exit(555);
}
}

View File

@@ -1,310 +0,0 @@
import { ERROR_CODE } from './Errors';
import { logger } from './Logger';
import cliProgress from 'cli-progress';
import WebSocket from 'ws';
export class DownloadManager {
// it's initalized in this.init()
private webSocket!: WebSocket;
private connected: boolean;
// NOTE: is there a way to fix the ETA? Can't get size nor ETA from aria that I can see
// we initialize this for each download
private progresBar!: cliProgress.Bar;
private completed: number;
private queue: Set<string>;
private index: number;
public constructor() {
this.connected = false;
this.completed = 0;
this.queue = new Set<string>();
this.index = 1;
if (!process.stdout.columns) {
logger.warn(
'Unable to get number of columns from terminal.\n' +
'This happens sometimes in Cygwin/MSYS.\n' +
'No progress bar can be rendered, however the download process should not be affected.\n\n' +
'Please use PowerShell or cmd.exe to run destreamer on Windows.'
);
}
}
/**
* MUST BE CALLED BEFORE ANY OTHER OPERATION
*
* Wait for an established connection between the webSocket
* and Aria2c with a 10s timeout.
* Then send aria2c the global config option if specified.
*/
public async init(port: number, options?: { [option: string]: string }): Promise<void> {
let socTries = 0;
const maxTries = 10;
let timer = 0;
const waitTime = 20;
const errorHanlder = async (err: WebSocket.ErrorEvent): Promise<void> => {
// we try for 10 sec to initialize a socket on the specified port
if (err.error.code === 'ECONNREFUSED' && socTries < maxTries) {
logger.debug(`[DownloadMangaer] trying webSocket init ${socTries}/${maxTries}`);
await new Promise(r => setTimeout(r, 1000));
this.webSocket = new WebSocket(`http://localhost:${port}/jsonrpc`);
this.webSocket.onerror = errorHanlder;
this.webSocket.onopen = openHandler;
socTries++;
}
else {
logger.error(err);
process.exit(ERROR_CODE.NO_CONNECT_ARIA2C);
}
};
const openHandler = (event: WebSocket.OpenEvent): void => {
this.connected = true;
logger.debug(`[DownloadMangaer] open event recived ${event}`);
logger.info('Connected to aria2 daemon!');
};
// create webSocket
this.webSocket = new WebSocket(`http://localhost:${port}/jsonrpc`);
this.webSocket.onerror = errorHanlder;
this.webSocket.onopen = openHandler;
// wait for socket connection
while (!this.connected) {
if (timer < waitTime) {
timer++;
await new Promise(r => setTimeout(r, 1000));
}
else {
process.exit(ERROR_CODE.NO_CONNECT_ARIA2C);
}
}
// setup messages handling
this.webSocket.on('message', (data: WebSocket.Data) => {
const parsed = JSON.parse(data.toString());
// print only messaged not handled during download
// NOTE: maybe we could remove this and re-add when the downloads are done
if (parsed.method !== 'aria2.onDownloadComplete' &&
parsed.method !== 'aria2.onDownloadStart' &&
parsed.method !== 'aria2.onDownloadError' &&
parsed.id !== 'getSpeed' &&
parsed.id !== 'addUrl' &&
parsed.id !== 'shutdown' &&
parsed.id !== 'getUrlForRetry') {
logger.info('[INCOMING] \n' + JSON.stringify(parsed, null, 4) + '\n\n');
}
});
if (options) {
logger.info('Now trying to send configs...');
this.setOptions(options);
}
this.webSocket.send(JSON.stringify({
jsonrpc: '2.0',
id: 'Destreamer',
method: 'aria2.getGlobalOption'
}));
logger.debug('[DownloadMangaer] Setup listener count on "message": ' + this.webSocket.listenerCount('message'));
}
public async close(): Promise<void> {
let exited = false;
let timer = 0;
const waitTime = 10;
this.webSocket.on('message', (data: WebSocket.Data) => {
const parsed = JSON.parse(data.toString());
if (parsed.result === 'OK') {
exited = true;
logger.verbose('Aria2c shutdown complete');
}
});
this.webSocket.send(this.createMessage('aria2.shutdown', null, 'shutdown'));
this.webSocket.close();
while ((this.webSocket.readyState !== this.webSocket.CLOSED) || !exited) {
if (timer < waitTime) {
timer++;
await new Promise(r => setTimeout(r, 1000));
}
else {
throw new Error();
}
}
}
private initProgresBar(): void {
this.progresBar = new cliProgress.SingleBar({
barCompleteChar: '\u2588',
barIncompleteChar: '\u2591',
format: 'progress [{bar}] {percentage}% {speed} MB/s {eta_formatted}',
noTTYOutput: true,
notTTYSchedule: 3000,
// process.stdout.columns may return undefined in some terminals (Cygwin/MSYS)
barsize: Math.floor((process.stdout.columns || 30) / 3),
stopOnComplete: true,
hideCursor: true,
});
}
private createMessage(method: 'aria2.addUri', params: [[string]] | [[string], object], id?: string): string;
private createMessage(method: 'aria2.tellStatus', params: [[string]] | [string, object], id?: string): string;
private createMessage(method: 'aria2.changeOption', params: [string, object], id?: string): string;
private createMessage(method: 'aria2.changeGlobalOption', params: [{ [option: string]: string }], id?: string): string;
private createMessage(method: 'system.multicall', params: [Array<object>], id?: string): string;
// FIXME: I don't know how to properly implement this one that doesn't require params..
private createMessage(method: 'aria2.getGlobalStat', params?: null, id?: string): string;
private createMessage(method: 'aria2.shutdown', params?: null, id?: string): string;
private createMessage(method: string, params?: any, id?: string): string {
return JSON.stringify({
jsonrpc: '2.0',
id: id ?? 'Destreamer',
method: method,
// This took 40 mins just because I didn't want to use an if...so smart -_-
...(!!params && { params: params })
});
}
private createMulticallElement(method: string, params?: any): any {
return {
methodName: method,
// This took 40 mins just because I didn't want to use an if...so smart -_-
...(!!params && { params: params })
};
}
/**
* For general options see
* {@link https://aria2.github.io/manual/en/html/aria2c.html#aria2.changeOption here}.
* For single download options see
* {@link https://aria2.github.io/manual/en/html/aria2c.html#aria2.changeGlobalOption here}
*
* @param options object with key: value pairs
*/
private setOptions(options: { [option: string]: string }, guid?: string): void {
const message: string = guid ?
this.createMessage('aria2.changeOption', [guid, options]) :
this.createMessage('aria2.changeGlobalOption', [options]);
this.webSocket.send(message);
}
public downloadUrls(urls: Array<string>, directory: string): Promise<void> {
return new Promise(resolve => {
this.index = 1;
this.completed = 0;
// initialize the bar as a new one
this.initProgresBar();
let barStarted = false;
const handleResponse = (data: WebSocket.Data): void => {
const parsed = JSON.parse(data.toString());
/* I ordered them in order of (probable) times called so
that we don't check useless ifs (even if we aren't caring about efficency) */
// handle download completions
if (parsed.method === 'aria2.onDownloadComplete') {
this.queue.delete(parsed.params.pop().gid.toString());
this.progresBar.update(++this.completed);
/* NOTE: probably we could use setIntervall because reling on
a completed download is good in most cases (since the segments
are small and a lot, somany and frequent updates) BUT if the user
internet speed is really low the completed downalods come in
less frequently and we have less updates */
this.webSocket.send(this.createMessage('aria2.getGlobalStat', null, 'getSpeed'));
if (this.queue.size === 0) {
this.webSocket.off('message', handleResponse);
logger.debug('[DownloadMangaer] End download listener count on "message": ' + this.webSocket.listenerCount('message'));
resolve();
}
}
// handle speed update packages
else if (parsed.id === 'getSpeed') {
this.progresBar.update(this.completed,
{ speed: ((parsed.result.downloadSpeed as number) / 1000000).toFixed(2) });
}
// handle download errors
else if (parsed.method === 'aria2.onDownloadError') {
logger.error('Error while downloading, retrying...');
const errorGid: string = parsed.params.pop().gid.toString();
this.queue.delete(errorGid);
// FIXME: I don't know if it's fixed, I was not able to reproduce a fail reliably
this.webSocket.send(this.createMessage('aria2.tellStatus', [errorGid, ['files']], 'getUrlForRetry'));
}
else if (parsed.id === 'getUrlForRetry') {
const retryUrl = parsed.result.files[0].uris[0].uri;
const retryTitle = parsed.result.files[0].path;
this.webSocket.send(this.createMessage('aria2.addUri', [[retryUrl], { out: retryTitle }], 'addUrl'));
}
// handle url added to download list in aria
else if (parsed.id === 'addUrl') {
// if we recive array it's the starting list of downloads
// if it's a single string it's an error download being re-added
if (typeof parsed.result === 'string') {
this.queue.add(parsed.result.gid.toString());
}
else if (Array.isArray(parsed.result)) {
parsed.result.forEach((gid: string) =>
this.queue.add(gid.toString())
);
if (!barStarted) {
barStarted = true;
logger.debug(`[DownloadMangaer] Starting download queue size: ${this.queue.size}`);
this.progresBar.start(this.queue.size, 0, { speed: 0 });
}
}
}
};
// FIXME: terrible workaround for 'https://github.com/snobu/destreamer/issues/232#issuecomment-699642770' :/
this.webSocket.removeAllListeners('message');
this.webSocket.on('message', (data: WebSocket.Data) => {
const parsed = JSON.parse(data.toString());
if (parsed.method !== 'aria2.onDownloadComplete' &&
parsed.method !== 'aria2.onDownloadStart' &&
parsed.method !== 'aria2.onDownloadError' &&
parsed.id !== 'getSpeed' &&
parsed.id !== 'addUrl' &&
parsed.id !== 'shutdown' &&
parsed.id !== 'getUrlForRetry') {
logger.info('[INCOMING] \n' + JSON.stringify(parsed, null, 4) + '\n\n');
}
});
logger.debug('[DownloadMangaer] Start download listener count on "message": ' + this.webSocket.listenerCount('message'));
this.webSocket.on('message', data => handleResponse(data));
const paramsForDownload: Array<any> = urls.map(url => {
const title: string = (this.index++).toString().padStart(16, '0') + '.encr';
return this.createMulticallElement(
'aria2.addUri', [[url], { out: title, dir: directory }]);
});
this.webSocket.send(
this.createMessage('system.multicall', [paramsForDownload], 'addUrl')
);
});
}
}

View File

@@ -1,64 +1,51 @@
/* let's start our error codes up high so we
don't exit with the wrong message if other modules exit with some code */
export const enum ERROR_CODE {
UNHANDLED_ERROR = 1000,
UNHANDLED_ERROR,
ELEVATED_SHELL,
CANCELLED_USER_INPUT,
MISSING_FFMPEG,
OUTDATED_FFMPEG,
UNK_FFMPEG_ERROR,
INVALID_VIDEO_GUID,
NO_SESSION_INFO,
NO_ENCRYPTION,
ARIA2C_CRASH,
NO_CONNECT_ARIA2C,
NO_DAEMON_PORT,
MISSING_ARIA2
NO_SESSION_INFO
}
export const errors: {[key: number]: string} = {
[ERROR_CODE.UNHANDLED_ERROR]: 'Unhandled error or uncaught exception! \n' +
'Please check your download directory/directories and try again. \n' +
'If this keep happening please report it on github "https://github.com/snobu/destreamer/issues"',
export const errors: { [key: number]: string } = {
[ERROR_CODE.UNHANDLED_ERROR]: 'Unhandled error!\n' +
'Timeout or fatal error, please check your downloads directory and try again',
[ERROR_CODE.ELEVATED_SHELL]: 'Destreamer cannot run in an elevated (Administrator/root) shell. \n' +
'Please run in a regular, non-elevated window.',
[ERROR_CODE.ELEVATED_SHELL]: 'Destreamer cannot run in an elevated (Administrator/root) shell.\n' +
'Please run in a regular, non-elevated window.',
[ERROR_CODE.CANCELLED_USER_INPUT]: 'Input was cancelled by user',
[ERROR_CODE.CANCELLED_USER_INPUT]: 'Input was cancelled by user',
[ERROR_CODE.MISSING_FFMPEG]: 'FFmpeg is missing! Destreamer requires FFmpeg to merge videos',
[ERROR_CODE.MISSING_FFMPEG]: 'FFmpeg is missing!\n' +
'Destreamer requires a fairly recent release of FFmpeg to download videos',
[ERROR_CODE.MISSING_ARIA2]: 'Aria2c is missing! Destreamer requires Aria2c to download videos',
[ERROR_CODE.MISSING_FFMPEG]: 'The FFmpeg version currently installed is too old!\n' +
'Destreamer requires a fairly recent release of FFmpeg to download videos',
[ERROR_CODE.UNK_FFMPEG_ERROR]: 'Unknown FFmpeg error',
[ERROR_CODE.UNK_FFMPEG_ERROR]: 'Unknown FFmpeg error',
[ERROR_CODE.INVALID_VIDEO_GUID]: 'Unable to get video GUID from URL',
[ERROR_CODE.INVALID_VIDEO_GUID]: 'Unable to get video GUID from URL',
[ERROR_CODE.NO_SESSION_INFO]: 'Could not evaluate sessionInfo on the page',
[ERROR_CODE.NO_ENCRYPTION]: 'Could not extract the encryption info from the playlist',
[ERROR_CODE.ARIA2C_CRASH]: 'The Aria2c RPC server crashed with the previous message',
[ERROR_CODE.NO_CONNECT_ARIA2C]: 'Could not connect to Aria2c JSON-RPC WebSocket before timeout!',
[ERROR_CODE.NO_DAEMON_PORT]: 'Could not get a free port to use'
[ERROR_CODE.NO_SESSION_INFO]: 'Could not evaluate sessionInfo on the page'
};
export const enum CLI_ERROR {
MISSING_INPUT_ARG = 'You must specify a URLs source. \n' +
'Valid options are -i for one or more URLs separated by space or -f for input file. \n',
MISSING_INPUT_ARG = 'You must specify a URLs source. \n' +
'Valid options are -i for one or more URLs separated by space or -f for input file. \n',
INPUT_ARG_CONFLICT = 'Too many URLs sources specified! \n' +
'Please specify a single source, either -i or -f \n',
INPUT_ARG_CONFLICT = 'Too many URLs sources specified! \n' +
'Please specify a single source, either -i or -f \n',
INPUTFILE_WRONG_EXTENSION = 'The specified inputFile has the wrong extension \n' +
'Please make sure to use path/to/filename.txt when useing the -f option \n',
INPUTFILE_WRONG_EXTENSION = 'The specified inputFile has the wrong extension \n' +
'Please make sure to use path/to/filename.txt when useing the -f option \n',
INPUTFILE_NOT_FOUND = 'The specified inputFile does not exists \n'+
'Please check the filename and the path you provided \n',
INPUTFILE_NOT_FOUND = 'The specified inputFile does not exists \n' +
'Please check the filename and the path you provided \n',
INVALID_OUTDIR = 'Could not create the default/specified output directory \n' +
'Please check directory and permissions and try again. \n'
INVALID_OUTDIR = 'Could not create the default/specified output directory \n' +
'Please check directory and permissions and try again. \n'
}

View File

@@ -5,6 +5,8 @@ import { logger } from './Logger';
/**
* This file contains global destreamer process events
*
* @note SIGINT event is overridden in downloadVideo function
*
* @note function is required for non-packaged destreamer, so we can't do better
*/
export function setProcessEvents(): void {
@@ -19,16 +21,6 @@ export function setProcessEvents(): void {
logger.error({ message: msg, fatal: true });
});
process.on('SIGINT', signal => {
logger.error(signal);
process.exit(777);
});
process.on('uncaughtException', (err: Error) => {
logger.error(err);
process.exit(ERROR_CODE.UNHANDLED_ERROR);
});
process.on('unhandledRejection', (reason: {} | null | undefined) => {
if (reason instanceof Error) {
logger.error({ message: (reason as Error) });

View File

@@ -35,9 +35,6 @@ function customPrint (info: winston.Logform.TransformableInfo): string {
else if (info.level === 'verbose') {
return colors.cyan('\n[VERBOSE] ') + info.message;
}
else if (info.level === 'debug') {
return colors.magenta('\n[debug] ') + info.message;
}
return `${info.level}: ${info.message} - ${info.timestamp}`;
}

View File

@@ -9,10 +9,6 @@ import jwtDecode from 'jwt-decode';
import puppeteer from 'puppeteer';
type Jwt = {
[key: string]: any
}
export class TokenCache {
private tokenCacheFile = '.token_cache';
@@ -25,18 +21,24 @@ export class TokenCache {
const session: Session = JSON.parse(fs.readFileSync(this.tokenCacheFile, 'utf8'));
const [isExpiring, timeLeft] = this.isExpiring(session);
type Jwt = {
[key: string]: any
}
const decodedJwt: Jwt = jwtDecode(session.AccessToken);
if (isExpiring) {
const now: number = Math.floor(Date.now() / 1000);
const exp: number = decodedJwt['exp'];
const timeLeft: number = exp - now;
if (timeLeft < 120) {
logger.warn('Access token has expired! \n');
return null;
}
else {
logger.info(`Access token still good for ${Math.floor(timeLeft / 60)} minutes.\n`.green);
return session;
}
logger.info(`Access token still good for ${Math.floor(timeLeft / 60)} minutes.\n`.green);
return session;
}
public Write(session: Session): void {
@@ -48,23 +50,11 @@ export class TokenCache {
logger.info('Fresh access token dropped into .token_cachen \n'.green);
});
}
public isExpiring(session: Session): [boolean, number] {
const decodedJwt: Jwt = jwtDecode(session.AccessToken);
const timeLeft: number = decodedJwt['exp'] - Math.floor(Date.now() / 1000);
if (timeLeft < (5 * 60)) {
return [true, 0];
}
else {
return [false, timeLeft];
}
}
}
export async function refreshSession(url: string): Promise<Session> {
const videoId: string = url.split('/').pop() ?? process.exit(ERROR_CODE.INVALID_VIDEO_GUID);
const browser: puppeteer.Browser = await puppeteer.launch({
executablePath: getPuppeteerChromiumPath(),
@@ -80,7 +70,7 @@ export async function refreshSession(url: string): Promise<Session> {
const page: puppeteer.Page = (await browser.pages())[0];
await page.goto(url, { waitUntil: 'load' });
await browser.waitForTarget((target: puppeteer.Target) => target.url().endsWith('microsoftstream.com/'), { timeout: 150000 });
await browser.waitForTarget((target: puppeteer.Target) => target.url().includes(videoId), { timeout: 30000 });
let session: Session | null = null;
let tries = 1;

View File

@@ -6,7 +6,6 @@ export type Session = {
export type Video = {
// the following properties are all for the title template
title: string;
duration: string;
publishDate: string;
@@ -14,20 +13,15 @@ export type Video = {
author: string;
authorEmail: string;
uniqueId: string;
// the following properties are all the urls neede for the download
outPath: string;
totalChunks: number; // Abstraction of FFmpeg timemark
playbackUrl: string;
posterImageUrl: string;
captionsUrl?: string
// final filename, already sanitized and unique
filename: string;
// complete path to save the video
outPath: string;
}
/* NOTE: expand this template once we are all on board with a list
/* TODO: expand this template once we are all on board with a list
see https://github.com/snobu/destreamer/issues/190#issuecomment-663718010 for list*/
export const templateElements: Array<string> = [
'title',

View File

@@ -24,8 +24,6 @@ async function extractGuids(url: string, client: ApiClient): Promise<Array<strin
.then((response: AxiosResponse<any> | undefined) => response?.data.metrics.videos);
const result: Array<string> = [];
logger.error(videoNumber);
// Anything above $top=100 results in 400 Bad Request
// Use $skip to skip the first 100 and get another 100 and so on
for (let index = 0; index <= Math.floor(videoNumber / 100); index++) {
@@ -37,7 +35,7 @@ async function extractGuids(url: string, client: ApiClient): Promise<Array<strin
response?.data.value.map((item: any) => item.id)
);
result.push(...partial);
result.push(...partial);
}
return result;
@@ -193,29 +191,20 @@ export function checkOutDir(directory: string): boolean {
}
export async function getUrlsFromPlaylist(playlistUrl: string, session: Session): Promise<Array<string>> {
return await ApiClient.getInstance(session).callUrl(playlistUrl, 'get', null, 'text')
.then(res => (res?.data as string).split(/\r?\n/)
.filter(line => !(line.startsWith('#') || line === '')));
}
export function checkRequirements(): void {
try {
const copyrightYearRe = new RegExp(/\d{4}-(\d{4})/);
const ffmpegVer: string = execSync('ffmpeg -version').toString().split('\n')[0];
if (parseInt(copyrightYearRe.exec(ffmpegVer)?.[1] ?? '0') <= 2019) {
process.exit(ERROR_CODE.OUTDATED_FFMPEG);
}
logger.verbose(`Using ${ffmpegVer}\n`);
}
catch (e) {
process.exit(ERROR_CODE.MISSING_FFMPEG);
}
try {
const aria2Ver: string = execSync('aria2c --version').toString().split('\n')[0];
logger.verbose(`Using ${aria2Ver}\n`);
}
catch (e) {
process.exit(ERROR_CODE.MISSING_ARIA2);
}
}

View File

@@ -9,7 +9,6 @@ import { parse as parseDuration, Duration } from 'iso8601-duration';
import path from 'path';
import sanitizeWindowsName from 'sanitize-filename';
function publishedDateToString(date: string): string {
const dateJs: Date = new Date(date);
const day: string = dateJs.getDate().toString().padStart(2, '0');
@@ -36,11 +35,18 @@ function isoDurationToString(time: string): string {
}
export async function getVideosInfo(videoGuids: Array<string>,
session: Session, subtitles?: boolean): Promise<Array<Video>> {
function durationToTotalChunks(duration: string): number {
const durationObj: any = parseDuration(duration);
const hrs: number = durationObj.hours ?? 0;
const mins: number = durationObj.minutes ?? 0;
const secs: number = Math.ceil(durationObj.seconds ?? 0);
return (hrs * 60) + mins + (secs / 60);
}
export async function getVideoInfo(videoGuids: Array<string>, session: Session, subtitles?: boolean): Promise<Array<Video>> {
const metadata: Array<Video> = [];
let title: string;
let duration: string;
let publishDate: string;
@@ -48,17 +54,17 @@ export async function getVideosInfo(videoGuids: Array<string>,
let author: string;
let authorEmail: string;
let uniqueId: string;
const outPath = '';
let totalChunks: number;
let playbackUrl: string;
let posterImageUrl: string;
let captionsUrl: string | undefined;
const apiClient: ApiClient = ApiClient.getInstance(session);
/* See 'https://github.com/snobu/destreamer/pull/203' for API throttling mitigation */
/* TODO: change this to a single guid at a time to ease our footprint on the
MSS servers or we get throttled after 10 sequential reqs */
for (const guid of videoGuids) {
const response: AxiosResponse<any> | undefined =
await apiClient.callApi('videos/' + guid + '?$expand=creator', 'get');
@@ -76,6 +82,8 @@ export async function getVideosInfo(videoGuids: Array<string>,
uniqueId = '#' + guid.split('-')[0];
totalChunks = durationToTotalChunks(response?.data.media['duration']);
playbackUrl = response?.data['playbackUrls']
.filter((item: { [x: string]: string; }) =>
item['mimeType'] == 'application/vnd.apple.mpegurl')
@@ -111,14 +119,11 @@ export async function getVideosInfo(videoGuids: Array<string>,
author: author,
authorEmail: authorEmail,
uniqueId: uniqueId,
// totalChunks: totalChunks, // Abstraction of FFmpeg timemark
outPath: outPath,
totalChunks: totalChunks, // Abstraction of FFmpeg timemark
playbackUrl: playbackUrl,
posterImageUrl: posterImageUrl,
captionsUrl: captionsUrl,
filename: '',
outPath: '',
captionsUrl: captionsUrl
});
}
@@ -126,8 +131,7 @@ export async function getVideosInfo(videoGuids: Array<string>,
}
export function createUniquePaths(videos: Array<Video>, outDirs: Array<string>,
template: string, format: string, skip?: boolean): Array<Video> {
export function createUniquePath(videos: Array<Video>, outDirs: Array<string>, template: string, format: string, skip?: boolean): Array<Video> {
videos.forEach((video: Video, index: number) => {
let title: string = template;
@@ -151,14 +155,9 @@ export function createUniquePaths(videos: Array<Video>, outDirs: Array<string>,
const finalFileName = `${finalTitle}.${format}`;
const cleanFileName = sanitizeWindowsName(finalFileName, { replacement: '_' });
if (finalFileName !== cleanFileName) {
logger.warn(
`Not a valid Windows file name: "${finalFileName}"` +
'\nReplacing invalid characters with underscores to ' +
'preserve cross-platform consistency.');
logger.warn(`Not a valid Windows file name: "${finalFileName}".\nReplacing invalid characters with underscores to preserve cross-platform consistency.`);
}
video.filename = finalFileName;
video.outPath = path.join(outDirs[index], finalFileName);
});

View File

@@ -1,38 +1,32 @@
import { ApiClient } from './ApiClient';
import { argv, promptUser } from './CommandLineParser';
import { getDecrypter } from './Decrypter';
import { DownloadManager } from './DownloadManager';
import { argv } from './CommandLineParser';
import { ERROR_CODE } from './Errors';
import { setProcessEvents } from './Events';
import { logger } from './Logger';
import { getPuppeteerChromiumPath } from './PuppeteerHelper';
import { drawThumbnail } from './Thumbnail';
import { TokenCache, refreshSession} from './TokenCache';
import { TokenCache, refreshSession } from './TokenCache';
import { Video, Session } from './Types';
import { checkRequirements, parseInputFile, parseCLIinput, getUrlsFromPlaylist} from './Utils';
import { getVideosInfo, createUniquePaths } from './VideoUtils';
import { checkRequirements, ffmpegTimemarkToChunk, parseInputFile, parseCLIinput} from './Utils';
import { getVideoInfo, createUniquePath } from './VideoUtils';
import { spawn, execSync, ChildProcess } from 'child_process';
import cliProgress from 'cli-progress';
import fs from 'fs';
import isElevated from 'is-elevated';
import portfinder from 'portfinder';
import puppeteer from 'puppeteer';
import path from 'path';
import tmp from 'tmp';
import { ApiClient } from './ApiClient';
// TODO: can we create an export or something for this?
const m3u8Parser: any = require('m3u8-parser');
const { FFmpegCommand, FFmpegInput, FFmpegOutput } = require('@tedconf/fessonia')();
const tokenCache: TokenCache = new TokenCache();
const downloadManager = new DownloadManager();
export const chromeCacheFolder = '.chrome_data';
tmp.setGracefulCleanup();
async function init(): Promise<void> {
setProcessEvents(); // must be first!
setProcessEvents(); // must be first!
logger.level = argv.debug ? 'debug' : (argv.verbose ? 'verbose' : 'info');
if (argv.verbose) {
logger.level = 'verbose';
}
if (await isElevated()) {
process.exit(ERROR_CODE.ELEVATED_SHELL);
@@ -118,31 +112,28 @@ async function DoInteractiveLogin(url: string, username?: string): Promise<Sessi
tokenCache.Write(session);
logger.info('Wrote access token to token cache.');
logger.info("At this point Chromium's job is done, shutting it down... \n\n");
logger.info("At this point Chromium's job is done, shutting it down...\n");
await browser.close();
return session;
}
async function downloadVideo(videoGUIDs: Array<string>,
outputDirectories: Array<string>, session: Session): Promise<void> {
const apiClient = ApiClient.getInstance(session);
async function downloadVideo(videoGUIDs: Array<string>, outputDirectories: Array<string>, session: Session): Promise<void> {
logger.info('Downloading video info, this might take a while...');
const videos: Array<Video> = createUniquePaths (
await getVideosInfo(videoGUIDs, session, argv.closedCaptions),
outputDirectories, argv.outputTemplate ,argv.format, argv.skip
);
logger.info('Fetching videos info... \n');
const videos: Array<Video> = createUniquePath (
await getVideoInfo(videoGUIDs, session, argv.closedCaptions),
outputDirectories, argv.outputTemplate, argv.format, argv.skip
);
if (argv.simulate) {
videos.forEach(video => {
videos.forEach((video: Video) => {
logger.info(
'\nTitle: '.green + video.title +
'\nOutPath: '.green + video.outPath +
'\nPublished Date: '.green + video.publishDate + ' ' + video.publishTime +
'\nPublished Date: '.green + video.publishDate +
'\nPlayback URL: '.green + video.playbackUrl +
((video.captionsUrl) ? ('\nCC URL: '.green + video.captionsUrl) : '')
);
@@ -151,228 +142,136 @@ async function downloadVideo(videoGUIDs: Array<string>,
return;
}
logger.info('Trying to launch and connect to aria2c...\n');
/* FIXME: aria2Exec must be defined here for the scope but later on it's complaining that it's not
initialized even if we never reach line#361 if we fail the assignment here*/
let aria2cExec: ChildProcess;
let arai2cExited = false;
await portfinder.getPortPromise({ port: 6800 }).then(
async (port: number) => {
logger.debug(`[DESTREAMER] Trying to use port ${port}`);
// Launch aria2c
aria2cExec = spawn(
'aria2c',
['--pause=true', '--enable-rpc', '--allow-overwrite=true', '--auto-file-renaming=false', `--rpc-listen-port=${port}`],
{stdio: 'ignore'}
);
aria2cExec.on('exit', (code: number | null, signal: string) => {
if (code === 0) {
logger.verbose('Aria2c process exited');
arai2cExited = true;
}
else {
logger.error(`aria2c exit code: ${code}` + '\n' + `aria2c exit signal: ${signal}`);
process.exit(ERROR_CODE.ARIA2C_CRASH);
}
});
aria2cExec.on('error', (err) => {
logger.error(err as Error);
});
// init webSocket
await downloadManager.init(port, );
// We are connected
},
error => {
logger.error(error);
process.exit(ERROR_CODE.NO_DAEMON_PORT);
}
);
for (const video of videos) {
const masterParser = new m3u8Parser.Parser();
logger.info(`\nDownloading video no.${videos.indexOf(video) + 1} \n`);
for (const [index, video] of videos.entries()) {
if (argv.skip && fs.existsSync(video.outPath)) {
logger.info(`File already exists, skipping: ${video.outPath} \n`);
continue;
}
const [isSessionExpiring] = tokenCache.isExpiring(session);
if (argv.keepLoginCookies && isSessionExpiring) {
logger.info('Trying to refresh access token...');
session = await refreshSession('https://web.microsoftstream.com/');
apiClient.setSession(session);
if (argv.keepLoginCookies && index !== 0) {
logger.info('Trying to refresh token...');
session = await refreshSession('https://web.microsoftstream.com/video/' + videoGUIDs[index]);
ApiClient.getInstance().setSession(session);
}
masterParser.push(await apiClient.callUrl(video.playbackUrl).then(res => res?.data));
masterParser.end();
const pbar: cliProgress.SingleBar = new cliProgress.SingleBar({
barCompleteChar: '\u2588',
barIncompleteChar: '\u2591',
format: 'progress [{bar}] {percentage}% {speed} {eta_formatted}',
// process.stdout.columns may return undefined in some terminals (Cygwin/MSYS)
barsize: Math.floor((process.stdout.columns || 30) / 3),
stopOnComplete: true,
hideCursor: true,
});
// video playlist url
let videoPlaylistUrl: string;
const videoPlaylists: Array<any> = (masterParser.manifest.playlists as Array<any>)
.filter(playlist =>
Object.prototype.hasOwnProperty.call(playlist.attributes, 'RESOLUTION'));
logger.info(`\nDownloading Video: ${video.title} \n`);
logger.verbose('Extra video info \n' +
'\t Video m3u8 playlist URL: '.cyan + video.playbackUrl + '\n' +
'\t Video tumbnail URL: '.cyan + video.posterImageUrl + '\n' +
'\t Video subtitle URL (may not exist): '.cyan + video.captionsUrl + '\n' +
'\t Video total chunks: '.cyan + video.totalChunks + '\n');
if (videoPlaylists.length === 1 || argv.selectQuality === 10) {
videoPlaylistUrl = videoPlaylists.pop().uri;
}
else if (argv.selectQuality === 0) {
const resolutions = videoPlaylists.map(playlist =>
playlist.attributes.RESOLUTION.width + 'x' +
playlist.attributes.RESOLUTION.height
logger.info('Spawning ffmpeg with access token and HLS URL. This may take a few seconds...\n\n');
if (!process.stdout.columns) {
logger.warn(
'Unable to get number of columns from terminal.\n' +
'This happens sometimes in Cygwin/MSYS.\n' +
'No progress bar can be rendered, however the download process should not be affected.\n\n' +
'Please use PowerShell or cmd.exe to run destreamer on Windows.'
);
videoPlaylistUrl = videoPlaylists[promptUser(resolutions)].uri;
}
else {
let choiche = Math.round((argv.selectQuality * videoPlaylists.length) / 10);
if (choiche === videoPlaylists.length) {
choiche--;
}
logger.debug(`Video quality choiche: ${choiche}`);
videoPlaylistUrl = videoPlaylists[choiche].uri;
}
// audio playlist url
// TODO: better audio playlists parsing? With language maybe?
const audioPlaylists: Array<string> =
Object.keys(masterParser.manifest.mediaGroups.AUDIO.audio);
const audioPlaylistUrl: string =
masterParser.manifest.mediaGroups.AUDIO.audio[audioPlaylists[0]].uri;
// if (audioPlaylists.length === 1){
// audioPlaylistUrl = masterParser.manifest.mediaGroups.AUDIO
// .audio[audioPlaylists[0]].uri;
// }
// else {
// audioPlaylistUrl = masterParser.manifest.mediaGroups.AUDIO
// .audio[audioPlaylists[promptUser(audioPlaylists)]].uri;
// }
const videoUrls = await getUrlsFromPlaylist(videoPlaylistUrl, session);
const audioUrls = await getUrlsFromPlaylist(audioPlaylistUrl, session);
const videoDecrypter = await getDecrypter(videoPlaylistUrl, session);
const audioDecrypter = await getDecrypter(videoPlaylistUrl, session);
const headers: string = 'Authorization: Bearer ' + session.AccessToken;
if (!argv.noExperiments) {
await drawThumbnail(video.posterImageUrl, session);
}
// video download
const videoSegmentsDir = tmp.dirSync({
prefix: 'video',
tmpdir: path.dirname(video.outPath),
unsafeCleanup: true
const ffmpegInpt: any = new FFmpegInput(video.playbackUrl, new Map([
['headers', headers]
]));
const ffmpegOutput: any = new FFmpegOutput(video.outPath, new Map([
argv.acodec === 'none' ? ['an', null] : ['c:a', argv.acodec],
argv.vcodec === 'none' ? ['vn', null] : ['c:v', argv.vcodec],
['n', null]
]));
const ffmpegCmd: any = new FFmpegCommand();
const cleanupFn: () => void = () => {
pbar.stop();
if (argv.noCleanup) {
return;
}
try {
fs.unlinkSync(video.outPath);
}
catch (e) {
// Future handling of an error (maybe)
}
};
pbar.start(video.totalChunks, 0, {
speed: '0'
});
logger.info('\nDownloading video segments \n');
await downloadManager.downloadUrls(videoUrls, videoSegmentsDir.name);
// audio download
const audioSegmentsDir = tmp.dirSync({
prefix: 'audio',
tmpdir: path.dirname(video.outPath),
unsafeCleanup: true
});
logger.info('\nDownloading audio segments \n');
await downloadManager.downloadUrls(audioUrls, audioSegmentsDir.name);
// subs download
// prepare ffmpeg command line
ffmpegCmd.addInput(ffmpegInpt);
ffmpegCmd.addOutput(ffmpegOutput);
if (argv.closedCaptions && video.captionsUrl) {
logger.info('\nDownloading subtitles \n');
await apiClient.callUrl(video.captionsUrl, 'get', null, 'text')
.then(res => fs.writeFileSync(
path.join(videoSegmentsDir.name, 'CC.vtt'), res?.data));
const captionsInpt: any = new FFmpegInput(video.captionsUrl, new Map([
['headers', headers]
]));
ffmpegCmd.addInput(captionsInpt);
}
logger.info('\n\nMerging and decrypting video and audio segments...\n');
ffmpegCmd.on('update', async (data: any) => {
const currentChunks: number = ffmpegTimemarkToChunk(data.out_time);
const cmd = (process.platform == 'win32') ? 'copy /b *.encr ' : 'cat *.encr > ';
pbar.update(currentChunks, {
speed: data.bitrate
});
execSync(cmd + `"${video.filename}.video.encr"`, { cwd: videoSegmentsDir.name });
const videoDecryptInput = fs.createReadStream(
path.join(videoSegmentsDir.name, video.filename + '.video.encr'));
const videoDecryptOutput = fs.createWriteStream(
path.join(videoSegmentsDir.name, video.filename + '.video'));
const decryptVideoPromise = new Promise(resolve => {
videoDecryptOutput.on('finish', resolve);
videoDecryptInput.pipe(videoDecrypter).pipe(videoDecryptOutput);
// Graceful fallback in case we can't get columns (Cygwin/MSYS)
if (!process.stdout.columns) {
process.stdout.write(`--- Speed: ${data.bitrate}, Cursor: ${data.out_time}\r`);
}
});
execSync(cmd + `"${video.filename}.audio.encr"`, {cwd: audioSegmentsDir.name});
const audioDecryptInput = fs.createReadStream(
path.join(audioSegmentsDir.name, video.filename + '.audio.encr'));
const audioDecryptOutput = fs.createWriteStream(
path.join(audioSegmentsDir.name, video.filename + '.audio'));
process.on('SIGINT', cleanupFn);
const decryptAudioPromise = new Promise(resolve => {
audioDecryptOutput.on('finish', resolve);
audioDecryptInput.pipe(audioDecrypter).pipe(audioDecryptOutput);
// let the magic begin...
await new Promise((resolve: any) => {
ffmpegCmd.on('error', (error: any) => {
cleanupFn();
logger.error(`FFmpeg returned an error: ${error.message}`);
process.exit(ERROR_CODE.UNK_FFMPEG_ERROR);
});
ffmpegCmd.on('success', () => {
pbar.update(video.totalChunks); // set progress bar to 100%
logger.info(`\nDownload finished: ${video.outPath} \n`);
resolve();
});
ffmpegCmd.spawn();
});
await Promise.all([decryptVideoPromise, decryptAudioPromise]);
logger.info('Decrypted!\n');
logger.info('Merging video and audio together...\n');
const mergeCommand = (
// add video input
`ffmpeg -i "${path.join(videoSegmentsDir.name, video.filename + '.video')}" ` +
// add audio input
`-i "${path.join(audioSegmentsDir.name, video.filename + '.audio')}" ` +
// add subtitles input if present and wanted
((argv.closedCaptions && video.captionsUrl) ?
`-i "${path.join(videoSegmentsDir.name, 'CC.vtt')}" ` : '') +
// copy codec and output path
`-c copy "${video.outPath}"`
);
logger.debug('[destreamer] ' + mergeCommand);
execSync(mergeCommand, { stdio: 'ignore' });
logger.info('Done! Removing temp files...\n');
videoSegmentsDir.removeCallback();
audioSegmentsDir.removeCallback();
logger.info(`Video no.${videos.indexOf(video) + 1} downloaded!!\n\n`);
process.removeListener('SIGINT', cleanupFn);
}
logger.info('Exiting, this will take some seconds...');
logger.debug('[destreamer] closing downloader socket');
await downloadManager.close();
logger.debug('[destreamer] closed downloader. Waiting aria2c deamon exit');
let tries = 0;
while (!arai2cExited) {
if (tries < 10) {
tries++;
await new Promise(r => setTimeout(r, 1000));
}
else {
aria2cExec!.kill('SIGINT');
}
}
logger.debug('[destreamer] stopped aria2c');
return;
}
async function main(): Promise<void> {
await init(); // must be first
const session: Session = tokenCache.Read() ??
await DoInteractiveLogin('https://web.microsoftstream.com/', argv.username);
let session: Session;
// eslint-disable-next-line prefer-const
session = tokenCache.Read() ?? await DoInteractiveLogin('https://web.microsoftstream.com/', argv.username);
logger.verbose('Session and API info \n' +
'\t API Gateway URL: '.cyan + session.ApiGatewayUri + '\n' +
@@ -390,13 +289,12 @@ async function main(): Promise<void> {
[videoGUIDs, outDirs] = await parseInputFile(argv.inputFile!, argv.outputDirectory, session);
}
logger.verbose('List of videos and corresponding output directory \n' +
logger.verbose('List of GUIDs and corresponding output directory \n' +
videoGUIDs.map((guid: string, i: number) =>
`\thttps://web.microsoftstream.com/video/${guid} => ${outDirs[i]} \n`).join(''));
// fuck you bug, I WON!!!
await downloadVideo(videoGUIDs, outDirs, session);
downloadVideo(videoGUIDs, outDirs, session);
}