In this blog I will be creating a simple node CLI tool for downloading Youtube videos with an option to automatically convert them to a specified audio or video format. Among others, this guide will cover the creation of global node modules and the usage of Typescript.

The code for this project can be found at https://github.com/learningdollars/martas-youtubedownloader.

Prerequisites

To start the project, you need to have Node.js  and npm installed. All the available install options are listed  here.

Part 1 – Project Setup

I will initialize the project by running npm init after which a series of inputs will be requested. Here, I will name my project “yt-converter”, add a short description and keywords. To skip this part and use the default values, run npm init with the -y option. After finishing, package.json file will be generated that will look like this:

{
  "name": "yt-converter",
  "version": "1.0.0",
  "description": "A simple tool for downloading youtube videos and converting them to a specified format.",
  "main": "index.js",
  "scripts": {
    "test": "echo \"No tests specified.\" && exit 0",
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/learningdollars/martas-project.git"
  },
  "keywords": [
    "youtube",
    "ffmpeg",
    "typescript",
    "node"
  ],
  "author": "martas",
  "license": "ISC",
  "bugs": {
    "url": "https://github.com/learningdollars/martas-project/issues"
  },
  "homepage": "https://github.com/learningdollars/martas-project#readme",
  "dependencies": {
  }
}

The next step is the installation of development dependencies:

npm install ts-node-dev typescript --save-dev

I’ll update package.json scripts field with the following lines:

"scripts": {
    "test": "echo \"No tests specified.\" && exit 0",
    "start": "node ./dist",
    "build": "npx tsc",
    "watch": "tsnd --respawn ./src"
  },

The build command runs the typescript compiler and the watch command runs the development server. To be able to run typescript compiler we will need tsconfig.json file, created by running: tsc --init or npx tsc --init. I will replace the default file content with the following:

{
  "exclude": [
    "tests",
    "node_modules",
    "dist"
  ],
  "compilerOptions": {
    /* Basic Options */
    "target": "es5",
    "resolveJsonModule":true,
    /* Specify ECMAScript target version: 'ES3' (default), 'ES5', 'ES2015', 'ES2016', 'ES2017','ES2018' or 'ESNEXT'. */
    "module": "commonjs",
    /* Specify module code generation: 'none', 'commonjs', 'amd', 'system', 'umd', 'es2015', or 'ESNext'. */
    "lib": [
      "es2015"
    ],
    /* Specify library files to be included in the compilation. */
    "declaration": true,                   /* Generates corresponding '.d.ts' file. */
    "outDir": "./dist",
    /* Strict Type-Checking Options */
    "strict": true,
    /* Enable all strict type-checking options. */
    "noImplicitAny": true,
    /* Additional Checks */
    "noUnusedLocals": true,
    /* Report errors on unused locals. */
    "noUnusedParameters": true,
    "esModuleInterop": true /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */
  },
}

In the last step of this section, I will create ./src/index.ts file as an entry point and add a .gitignore file:

/node_modules
/dist

Part 2 – Project Walkthrough

I will start the development by creating YtConverter class and adding the following code to index.ts file:

import YtConverter from "./YtConverter"

export function init(urls?: Array<string&gt;, dir?: string, convert?: boolean, format?: string) {
    new YtConverter().init(urls, dir, convert, format).catch((err: Error) =&gt; {
        console.error("ERROR", err)
        process.exit(1)
    })
}
export default class YtConverter {
    /**
     * @param  {Array<string&gt;} urls
     * - array of video urls to download
     * @param  {string} dir - optional
     * - destination directory - default /home/{user}/videos
     * @param  {boolean} convert - optional
     *  - should the videos be converted - default: false
     * @param  {string} format - optional
     *  - format to convert into - default: "mp3"
     *  Initializes video download / conversion
     */
    async init(urls?: Array<string&gt;, dir?: string, convert?: boolean, format?: string, update?: boolean) {
    }
}

YtConverter.init function will take 5 (optional) paramaters:

  • urls – and array of Youtube video URLs
  • dir – download destination directory
  • convert – boolean value determining weather the video should be converted
  • format – format to convert into
  • update – should ffmpeg be checked for updates (ffmpeg binary will be downloaded later in this guide)

You can start the development server by running npm run watch. For now, multiple errors will be displayed in the console, as all the parameters of the YtConverter.init function are not used and the "noUnusedParameters" rule is set to true in tsconfig.json file. Also, Node.js process and console objects will produce type errors, that can be solved by installing type definitions for Node.js: npm install @types/node --save-dev.

Next, I’ll add a new class Args in Args.ts file that will parse command-line arguments and set them as instance fields. Here I will need yargs module and its types, so I will run: npm install yargs @types/yargs .

import yargs from "yargs"

export default class Args {
    private _urls!: Array<string | number&gt;
    private _dir!: string
    private _convert!: boolean
    private _format!: string
    private _update!: boolean

    constructor() {
        this.parseCliArgs()
    }
    parseCliArgs() {
        const args = yargs
            .option('url', {
                alias: "u",
                describe: "video url to download",
                array: true,
            })
            .option('dir', {
                alias: "d",
                default: `/home/${require("os").userInfo().username}/videos`,
                describe: "destination directory",
                string: true
            })
            .option('convert', {
                alias: "c",
                default: false,
                describe: "convert to mp3",
                boolean: true
            })
            .option('format', {
                alias: "f",
                default: "mp3",
                describe: "format to convert into",
                string: true
            })
            .option('update-ffmpeg', {
                alias: "uf",
                default: false,
                describe: "update ffmpeg",
                boolean: true
            })
            .alias("h", "help")
            .help('help')
            .argv
        if (args.url)
            this._urls = args.url
        this._dir = args.dir
        this._convert = args.convert
        this._format = args.format
        this._update = args["update-ffmpeg"]
    }
    /**
     * @param  {Array<string&gt;} urls optional - an array of urls
     * @param  {string} dir? - target directory
     * @param  {boolean} convert? should the video file be converted
     * @param  {string} format? format to convert into
     * @param  {boolean} update? update ffmpeg
     * 
     * Replaces CLI arguments with parameters passed from YtConverter.init funtion
     */
    setArgs(urls?: Array<string&gt;, dir?: string, convert?: boolean, format?: string, update?: boolean) {
        if (urls)
            this._urls = urls
        if (dir)
            this._dir = dir
        if (convert)
            this._convert = convert
        if (format)
            this._format = format
        if (update)
            this._update = update
        if (!this._urls || this._urls.length === 0)
            throw new Error("Url argument is required!")
    }

    get urls() {
        return this._urls
    }

    get dir() {
        return this._dir
    }

    get convert() {
        return this._convert
    }

    get format() {
        return this._format
    }

    get update() {
        return this._update
    }
}

In the Args.parseCliArgs function I am defining usage, type and the default values for each field. The only required argument will be urls, but I wont add { demand: true } as I want this module to be usable locally. Args.setArgs function will set the values to instance fields, after checking for locally passed parameters.

Next, I will add Instances class that will initialize and hold all the used instances. For now, this class will only initialize Args class and it looks like this:

import Args from "./Args";

export default class Instances {
    private _args: Args

    constructor() {
        this._args = new Args()
    }

    get args(): Args {
        return this._args
    }
}

To YtConverter.init function I will add the following lines that will pass the function parameters to this._args instance, check the validity of urls argument, and log the beginning of the process:

async init(urls?: Array<string&gt;, dir?: string, convert?: boolean, format?: string, update?: boolean) {
        this._args.setArgs(urls, dir, convert, format, update)
        if (!Array.isArray(this._args.urls)) {
            throw new Error("The first argument must be an array!")
        }
        process.stdout.write(`\nDownloading ${this._args.urls.length} videos to directory ${this._args.dir}`)
    }

Next, I will add the Downloader class, and install all its required dependencies: npm install youtube-dl @types/youtube-dl humanize. As the last module doesn’t have a type definition module, I’ll have to create it on my own. I’ll create types directory on the root directory and add humanize.d.ts file:

declare module "humanize" {
    export function filesize(size: number): string
}

As the filesize function is the only one I’ll use, it will be the only one whose type I’ll define.

The Downloader class will need a few helper functions so I will add a new Utils class with the following content:

import fs from "fs"
import shell from "shelljs"

export default class Utils {
    private _interval!: NodeJS.Timeout
    ffPath!: string
    /**
     * @param  {string} dir directory path
     * Checks if the directory exists, and if not, it creates it.
     */
    async checkDir(dir: string) {
        return new Promise((resolve) => {
            if (!fs.existsSync(dir)) {
                shell.mkdir("-p", dir)
                resolve()
            }
            resolve()
        })
    }
    /**
     * @param  {boolean} run - start of stop loader
     */
    loader(run: boolean) {
        if (this._interval && !run) {
            clearInterval(this._interval)
        }
        else {
            this._interval = setInterval(() => {
                process.stdout.write(" .")
            }, 200)
        }
    }
}

The checkDir function will check if the target directory exists, and if it doesn’t, it will create it. The loader function will print dots in the console during longer processes (e.g. video download requests) as a simple UX improvement.

Downloader class now looks like this:

import youtubedl from "youtube-dl";
import fs from "fs"
import path from "path"
import Utils from "./Utils";
import humanize from "humanize"
import ProcessExt from "../types/processExt";

export default class Downloader {
    private _utils: Utils
    constructor(utils: Utils) {
        this._utils = utils
    }
    /**
     * @param  {string} url - video url
     *  - Get video information from youtube
     */
    async getVideoInfo(url: string): Promise<youtubedl.Info&gt; {
        // check if url is a video url
        if (!url.match("/watch?"))
            throw new Error("Invalid video url")
        return new Promise((res, rej) =&gt; {
            return youtubedl.getInfo(url, (err: Error, info: any) =&gt; {
                if (err)
                    rej(err)
                else {
                    res(info)
                }
            })
        })
    }

    /**
     * @param  {string} url - video url
     * @param  {string} dir - destination directory
     * @returns {object} video filename and directory.
     */
    async downloadVideos(url: string, dir: string): Promise<{ filename: string, dir: string }&gt; {
        this._utils.loader(true)

        return new Promise(async (resolve, reject) =&gt; {
            let processExt: ProcessExt = process
            let video: youtubedl.Youtubedl
            let videoInfo: youtubedl.Info
            let position = 0;
            let videoSize = 0;
            let filename:string

            try {
                videoInfo = await this.getVideoInfo(url)
            }
            catch (err) {
                reject(err)
                return
            }
            if (!videoInfo) {
                throw new Error(`Unable to get video info from url ${url}`)
            }
            filename = videoInfo._filename

            await this._utils.checkDir(dir)
            video = youtubedl(url,
                ['--format=18'],
                { cwd: dir });

            video.on('info', (info: youtubedl.Info) =&gt; {
                this._utils.loader(false)
                videoSize = info.size
                console.log(`\nDownloading ${info._filename}`)
                console.log(`Size: ${humanize.filesize(info.size)}`);
            });

            video.on('data', (chunk: any) =&gt; {
                position += chunk.length;
                if (videoSize) {
                    // calculate video download percentage
                    var percent = (position / videoSize * 100).toFixed(2);
                    if (processExt.stdout.cursorTo &amp;&amp; processExt.stdout.clearLine) {
                        processExt.stdout.cursorTo(0);
                        processExt.stdout.clearLine(1);
                        process.stdout.write(percent + '%');
                    }
                }
            });
            video.on('end', async () =&gt; {
                process.stdout.write(' - DONE\n');
                resolve({ filename, dir })
                this._utils.loader(false)
            })

            video.pipe(fs.createWriteStream(`${path.join(dir, filename)}`));
        })
    }
}

It consists of getVideoInfo and downloadVideos functions. The former fetches the video information using youtubedl module, and the latter downloads the video if the video information was successfully fetched and if the target directory was created.

By calling youtubedl module, video stream is returned that will be piped to the writable stream of a destination file. To the video stream I’ll attach event listeners with handler functions. These functions will simply compare the downloaded size with full video size and log progress in the form of percentages.

In addition, I’ll extend the process.stdout interface to include cursorTo and clearLine functions, as they are not defined in NodeJS.process.stdout type definition by default.

New classes Utils and Downloader will be instantiated in the Instances class, so I’ll add them there:

import Args from "./Args";
import Downloader from "./Downloader";
import Utils from "./Utils";

export default class Instances {
    private _args: Args
    private _utils: Utils
    private _downloader!: Downloader

    constructor() {
        this._args = new Args()
        this._utils = new Utils()
    }

    get args(): Args {
        return this._args
    }
    get utils(): Utils {
        return this._utils
    }

    get downloader(): Downloader {
        if (!this._downloader) {
            this._downloader = new Downloader(this.utils)
        }
        return this._downloader
    }
}

I’ll call downloder.downloadVideos function from YtConverter.init function:

    for (let url of this._args.urls) {
          await this._downloader.downloadVideos(url.toString(), this._args.dir).catch(err => { throw err })
        }

Now, download functionality is done, so if I add a call to init function (init()) in index.ts, I will be able to download a video by running:

node ./dist -u [some-yt-video-url]

and by adding -d [destination-dir] I can customize the target directory.

The terminal output of this action looks like this:

The next step is to add a converting functionality. For transcoding video files in other formats, I’ll use ffmpeg, a command-line multimedia software, able to transcode from and to, all the existing media formats. To be able to use ffmpeg, I’ll have to download the binary from its repository using the API endpoint. Additionally, I’ll add an option to update ffmpeg, in the case of new releases.

To start, I’ll add a new directory ffmpeg-downloader with index.ts and ffInfo.json files. The former will hold all the downloading logic and the latter will hold the path to the binary file and the current version number. I’ll start by adding the following content to ffmpeg-downloader/index.ts file:

const LATEST = `http://ffbinaries.com/api/v1/version/latest`

import request, { Response } from "request"
import progress from "request-progress"
import fs from "fs"
import path from "path"
import Utils from "../Utils";
import ffInfo from "./ffInfo.json"

export default class FfmpegDownloader {

    utils: Utils
    version!: string
    url!: string
    platform!: string
    ffPath!: string

    constructor(utils: Utils) {
        this.utils = utils
    }
   
    async init() {
        const { update, version } = await this.checkUpdate()
        if (!update) {
            console.log(`Ffmpeg already on the latest version: ${version} `)
            return
        }
        await this.downloadVersion()
    }
    /**
     * @returns {object} update: should ffmpeg be updated; version: current ffmpeg version
     * 
     * - Check if ffmpeg should be updated
     */
    async checkUpdate(): Promise<{ update: boolean, version: string }&gt; {
        const ffVersion = ffInfo.ffVersion
        return new Promise((resolve, reject) =&gt; {
            request.get(LATEST, { json: true }, (err: any, _: Response, body: any) =&gt; {
                if (err) {
                    reject(err)
                    return
                }
                if (body.version === ffVersion) {
                    resolve({ update: false, version: ffVersion })
                    return
                }
                this.version = body.version
                const platform = this.utils.getPlatform()
                if (!platform) {
                    throw new Error("Unable to download ffmpeg")
                }
                this.url = body.bin[platform].ffmpeg
                this.platform = platform
                resolve({ update: true, version: this.version })
            })
        })
    }
  
    /**
     * @returns Promise<void&gt;
     * 
     * - Downloads ffmpeg.
     */
    async downloadVersion():Promise<void&gt; {
        const ffDir = path.join(__dirname, `../../ffmpeg/${this.platform}/${this.version}`)
        await this.utils.checkDir(ffDir)
        this.ffPath = path.join(ffDir, "ffmpeg")
        const zipPath = this.ffPath + ".zip"
        // If platform is windows, add .exe extension and replace path slashes.
        if (this.utils.getPlatform() === "windows-64") {
            this.ffPath += ".exe"
            this.ffPath = this.ffPath.replace(/\\/g, "/")
        }
        fs.writeFileSync(path.join(__dirname, "./ffInfo.json"), Buffer.from(`{ "ffPath": "${this.ffPath}","ffVersion":"${this.version}"}`))
        // Download ffmpeg
        return new Promise((resolve, reject) =&gt; {
            process.stdout.write(`Downloading ffmpeg v${this.version}`)
            this.utils.loader(true)
            progress(request.get(this.url, { json: true }))
                .on("end", async () =&gt; {
                    this.utils.loader(false)
                    process.stdout.write(`\nExtracting ffmpeg v${this.version}`)
                    await this.utils.extract(zipPath, ffDir)
                    resolve()
                })
                .on("error", () =&gt; {
                    reject("An error occured")
                    return
                })
                .pipe(fs.createWriteStream(zipPath))
        })
    }
}

For this to work, I have to install additional dependencies: npm install request request-progress @types/request and add a type definition for request-progress module in types/requestProgress.d.ts file:

declare module "request-progress"

FfmpegDownloader class uses 2 additional helper functions: getPlatform and extract, so I’ll add them to the Utils class. getPlatform function checks for the system platform using Node.js “os” module so the correct binary could be downloaded. As the binary is downloaded in the .zip format, extract extracts the file in the same directory, and deletes the original file.

    /**
     * @param  {string} path target file path
     * @param  {string} dir destination directory
     * @returns Promise
     */
    async extract(path: string, dir: string): Promise<void&gt; {
        return new Promise((resolve, reject) =&gt; {
            this.loader(true)
            extract(path, { dir: dir, defaultFileMode: 777 }, (err: Error | undefined) =&gt; {
                if (err) {
                    reject(err)
                    this.loader(false)
                    return
                }
                fs.unlink(path, (err: NodeJS.ErrnoException | null) =&gt; {
                    if (err) {
                        reject(err)
                        return
                    }
                    resolve()
                    this.loader(false)
                })
            })
        })
    }

    /**
     * @returns string | null
     */
    getPlatform(): string | null {
        switch (os.platform()) {
            case ("win32"): {
                return "windows-64"
            }
            case ("linux"): {
                if (os.arch() === "x64")
                    return "linux-64"
                else if (os.arch() === "x32")
                    return "linux-32"
            }
            case ("darwin"): {
                return "osx-64"
            }
            default: return null
        }
    }

Now ffmpeg downloading functionality is done. If I run: npm run build && node dist/index.js -u https://www.youtube.com/watch\?v=zMqP1sIheeM -d ./video --uf , ffmpeg will be downloaded before file download. If I run the command again, the download will be skipped as the versions are the same.

The last thing left to do is the use the ffmpeg binary to transcode the downloaded videos. To accomplish this, I’ll add a new Converter class in Converter.ts file:

import fs from "fs"
import path from "path"
import Utils from "./Utils";
import { exec } from "child_process"
import ProcessExt from "../types/processExt";
import { ExecFileError } from "../types/execFileError";

export default class Converter {
    private _utils: Utils
    constructor(utils: Utils) {
        this._utils = utils
    }
    /**
     * @param  {string} file - filename to convert
     * @param  {string} dir - destination directory
     * @param  {string} format - format to convert into
     * - Converts a file into a given format.
     */
    async convert(file: string, dir: string, format: string) {
        const fileArr = file.split(".")
        let ext = fileArr.pop() || "mp4"
        // Read ffmpeg binary path from ffInfo.json
        const ffPath = JSON.parse(fs.readFileSync(path.join(__dirname, "./ffmpeg-downloader/ffInfo.json")).toString()).ffPath

        const filename = fileArr.join(".")
        console.log(`\nConverting ${file} to ${filename}.${format}`)
        await this._utils.checkDir(`${dir}/${format}`)
        let cmd = `${ffPath} -y -i ${dir}/"${filename.concat(".", ext)}" ${dir}/"${format}/${filename}.${format}"`
        return new Promise((resolve) =&gt; {
            let duration = 0
            const converter = exec(cmd,
                (err: ExecFileError | null) =&gt; {
                    if (err) {
                        if (err.code &amp;&amp; err.code === "ENOENT") {
                            throw new Error("ffmpeg must be installed!")
                        }
                        throw err
                    }
                })
            if (!converter.stderr) {
                throw new Error("Invalid commmand.")
            }
            converter.stderr.on("end", () =&gt; {
                resolve()
            })

            converter.stderr.on("data", (data: any) =&gt; {
                const durationIndex = data.search("Duration")

                if (durationIndex !== -1) {
                    // find the duration substring in ffmpeg output
                    let durData = data.substring(durationIndex, durationIndex + 18).split(" ")[1].trim()
                    duration = this._utils.getDuration(durData)
                }
                else {
                    let processE: ProcessExt = process
                    const timeIndex = data.search("time=")
                    if (timeIndex === -1)
                        return
                    const timeData = data.substring(timeIndex, timeIndex + 13).split("=")[1].trim()
                    const time = this._utils.getDuration(timeData)
                    if (duration === NaN || time === NaN)
                        return
                    const percentage = ((time / duration) * 100) <= 100 ? ((time / duration) * 100).toFixed(2) : "100.00"
                    if (processE.stdout.cursorTo &amp;&amp; processE.stdout.clearLine) {
                        processE.stdout.cursorTo(0);
                        processE.stdout.clearLine(1);
                        processE.stdout.write(percentage + "%");
                    }
                    if (percentage == "100.00")
                        process.stdout.write(" - DONE\n");
                }
            })
        })
    }
}

For this to work, I’ll need to add getDuration function to Utils class:

  /**
    * @param  {string} data string in format hh:mm:ss
    * @returns {number} duration in miliseconds
    */
    getDuration(data: string): number {
        let dataArr = data.split(":")
        let duration = 0
        if (dataArr.length !== 3)
            return NaN
        dataArr.forEach((d: string, i: number) => {
            switch (i) {
                case (0): duration += parseInt(d) * 3600 * 1000
                    break;
                case (1): duration += parseInt(d) * 60 * 1000
                    break;
                case (2): duration += parseInt(d) * 1000
            }
        })
        return duration
    }

This function will attempt to parse duration string from ffmpeg output in the format [hh:mm:ss] and return the duration in miliseconds. This data will be used to generate progress percentage that will be printed to the console.

Converter class consists of convert function that reads ffmpeg path and executes the binary with arguments provided through the command-line or as function parameters. exec function returns a child_process object that can be listened for events. I’m listening for “data” events to parse the duration string and log the progress percentage and the “end” event to resolve the Promise.

Next, I’ll add the converter to instances, import it in YtConverter class and run the converter if the convert argument is true:

  for (let url of this._args.urls) {
            const { filename, dir } = await this._downloader.downloadVideos(url.toString(), this._args.dir).catch(err => { throw err })
            if (this._args.convert)
                await this._converter.convert(filename, dir, this._args.format).catch((err: Error) => { throw err })
        }

All that is left to do now is to prepare this module for global installation.

Preparing for global installation

Create a new folder named bin inside the root directory of your module with an exec.js file in it with the following content:

#!/usr/bin/env node
require('../dist/index.js').init()

The first line (#!/usr/bin/env node) will allow npm to correctly generate an executable for this module.

The last thing to do is to register the package as global by adding the following to package.json:

 "bin": {
    "ytc": "./bin/exec.js"
  }

Here, I’m setting the path to the executable file which will be run with ytc command. Now this module can be installed with the -g option. You can test this by running in the root directory:

npm install -g .

The module is now ready. I’ll make an additional modification that will run ffmpegDownloader after the module is installed: I’ll add a small script in src/scripts/downloadFfmpeg.js with the following content:

import Instances from "../Instances";

const instances = new Instances()
instances.ffmpegDownloader.init()

For the script to run before the installation, I’ll add “postinstall” script in package.json.

 "postinstall": "tsc && node ./dist/scripts/downloadFFmpeg.js",

After the installation, this module can be run from the command-line:

ytc -u [video-url] -d [custom-directory] -c

For the list of all options, run:

ytc --help

The code for this project can be found at https://github.com/learningdollars/martas-youtubedownloader.