A few days ago, it was pointed to me that my website was featured on the S-Config Webrings page and I was super happy to find that out. However, I was very disappointed in myself upon reading something there about my site:

There are sections that break rather easily on his site as some of his media links back to YouTube which bars a person (depending on the country) from viewing

— S-config

As this was a very serious topic, especially since I’ve spoken about web accessibility five months ago, I’ve decided that something needs to be done. After trying to fiddle with the some Invidious installations here and there and playing cat and mouse for a few days when instances raise and fall faster than bitcoin prices, I decided that using Invidious will be an endless cat-and-mouse game that I don’t want to play, because it’s very time consuming and gives me no benefits. However, as I understand that the issues presented there are valid, so I decided to write my own Google-less Youtube-ish embeds.

1. Getting all the Youtube Embeds From My Site

Luckily, I’m smart and I use hugo’s built-in system of of shortcodes, so whenever I need to embed a clip, in a post or a quickie post, I’d just use the code

1
2

{{< youtube VIDEO_ID >}}
{{< youtube id="VIDEO_ID" title="Some Title" >}}

which in turn would render in the browser a simple piece of code, located in layouts/shortcodes/youtube.html

1
2
3
4
5
6
7
8
9

{{ $video := .Get 0 }}{{ if (.Get "id") }}{{ $video = .Get "id" }}{{end}}{{ $theVideo := split $video "?" }}{{ $title := "Youtube Video" }}{{ if (.Get "title") }}{{ $title = .Get "title" }}{{end}}
<div class="yt-container">
	<a title="load player" class="load-yt" target="_blank" href="http://www.youtube.com/watch?v={{ $video }}" data-id="{{ $video }}">
		<svg aria-hidden="true" class="icon-movies" viewBox="0 0 32 32" fill="currentColor"><use xlink:href="#icon-movies"></use></svg>
		<img src="{{ $.Site.Params.mediaURL }}/uploads/vi/{{ index ($theVideo) 0 }}.jpg">
	</a>
	<p class="pull-right"><a target="_blank" href="http://www.youtube.com/watch?v={{ $video }}">youtube</a> • video ID: {{ $video }}</p>
	<p>click image to load player</p>
</div>

The first line just defines the ID of the video to being used and an optional title, if I decide to use the fancier key=“value” variant of the shortcode and the variables makes sure both cases are taken care of. Then I’m just displaying the thumbnail photo with a cute icon on top, which on click loads the default Youtube embed iframe. If your javascript is off, then it just opens the video in a new tab/page. If you block youtube completely, you can copy the video ID displayed at the bottom and just paste it into your favourite Invidious instance.

Update

I had to split the $video string after the ? character in order to use it as the image thumbnail, as sometimes I’m linking videos to a certain timestamp (like yGXxawJdviE?t=17) and that would break the thumbnail display. I did that by simply using this snippet in hugo in the area where I’m displaying the video thumbnail:

1
2

{{ $theVideo := split $video "?" }}
{{ index ($theVideo) 0 }}

Additionally, I’m using Youtube as provider for trailers in the movies section, as well for music embeds on the records pages. These don’t use the shortcode per se, but they use a similar HTML and the Youtube IDs are added to each using a front-matter parameter, so I’ve just written a PHP script that searches all my files in the content/ directory and processes them. For the records, it’s a bit complex as it’s some arrays, so I’m just using thise snippet of code which is magic:

1
2
3

{{- range (where .Site.RegularPages.ByLastmod.Reverse "Type" "records") }}{{ range $v := .Params.video }}
	{{ . }}
{{ end }}{{ end }}

2. The Functionality

Javascript (jQuery in my case), with a graceful degradation of performance in case your browser doesn’t support Javascript or it’s disabled, case in which it just opens the page in a new tab or window. Additionally, the iframe embed is loading from youtube-nocookie.com, so you won’t get pushed any tracking cookies (Google’s word, don’t trust it!).

1
2
3
4
5
6
7

$('.load-yt').click(function(e){
	e.preventDefault();
	$elem =jQuery(this);
	$elemid =jQuery(this).attr('data-id');
	$('<div class="yt-vid"><iframe src="https://www.youtube-nocookie.com/embed/'+$elemid+'?rel=0" allowfullscreen="" width="100%" height="370" style="width: 100%; min-height: 370px;" frameBorder="0"></iframe></div>').insertAfter($elem);
	$elem.hide();
});

3. Downloading the Thumbnails

So as soon as I find a video ID, I check if I already have the thumbnail file. If it’s there, I just skip it. Otherwise, I try to download the HD version, named “maxresdefault.jpg”. If it returns a 404, I try with the lower version, “hqdefault.jpg”, which is working for most videos. Youtube has being using always the same naming structure for thumnails (documented here), but I needed to download only these URLs: https://img.youtube.com/vi/<insert-youtube-video-id-here>/maxresdefault.jpg and https://img.youtube.com/vi/<insert-youtube-video-id-here>/hqdefault.jpg.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


if(file_exists('uploads/vi/'.trim($line).'.jpg')) {
	echo "Thumb uploads/vi/".trim($line).".jpg ALREADY EXISTS\n";
} else {
	$picurl = 'http://i.ytimg.com/vi/'.trim($line).'/maxresdefault.jpg';
	$header = array();
	$header[] = 'Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'; 
	$header[] = 'Cache-Control: max-age=0'; 
	$header[] = 'Content-Type: text/html; charset=utf-8'; 
	$header[] = 'Transfer-Encoding: chunked'; 
	$header[] = 'Connection: keep-alive'; 
	$header[] = 'Keep-Alive: 300'; 
	$header[] = 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7'; 
	$header[] = 'Accept-Language: en-us,en;q=0.5'; 
	$header[] = 'Pragma:'; 
 	$ch = curl_init($picurl);
	$fp = fopen('uploads/vi/'.trim($line).'.jpg', 'wb');
	curl_setopt($ch, CURLOPT_FILE, $fp);
	curl_setopt($ch, CURLOPT_HEADER, 0);
	// curl_setopt($ch, CURLOPT_VERBOSE, true);
	curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'); 
	curl_setopt($ch, CURLOPT_HTTPHEADER, $header); 
	curl_setopt($ch, CURLOPT_REFERER, 'http://www.youtube.com'); 
	if(curl_exec($ch) === false) {
		echo 'Curl error: ' . curl_error($ch);
	} else {
		$http_status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
		if($http_status == 404) {
		 	$picurl2 = 'http://i.ytimg.com/vi/'.trim($line).'/hqdefault.jpg';
			$header = array();
			$header[] = 'Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'; 
			$header[] = 'Cache-Control: max-age=0'; 
			$header[] = 'Content-Type: text/html; charset=utf-8'; 
			$header[] = 'Transfer-Encoding: chunked'; 
			$header[] = 'Connection: keep-alive'; 
			$header[] = 'Keep-Alive: 300'; 
			$header[] = 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7'; 
			$header[] = 'Accept-Language: en-us,en;q=0.5'; 
			$header[] = 'Pragma:'; 
		 	$ch2 = curl_init($picurl2);
			$fp = fopen('uploads/vi/'.trim($line).'.jpg', 'wb');
			curl_setopt($ch2, CURLOPT_FILE, $fp);
			curl_setopt($ch2, CURLOPT_HEADER, 0);
			// curl_setopt($ch2, CURLOPT_VERBOSE, true);
			curl_setopt($ch2, CURLOPT_USERAGENT, 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'); 
			curl_setopt($ch2, CURLOPT_HTTPHEADER, $header); 
			curl_setopt($ch2, CURLOPT_REFERER, 'http://www.youtube.com'); 
			if(curl_exec($ch2) === false) {
				echo 'Curl error: ' . curl_error($ch);
			} else {
				echo "Thumb uploads/vi/".trim($line).".jpg created successfully from smaller\n";
			}
		} else {
			echo "Thumb uploads/vi/".trim($line).".jpg created successfully\n";
		}
	}
	curl_close($ch);
	fclose($fp);
}

The script output is simple to understand and can be extended, but for now it’s just enough:

1
2
3
4
5
6
7
8

Thumb `uploads/vi/xGg47V9VPL8.jpg` created successfully
Thumb `uploads/vi/TbbUEEIhhrI.jpg` created successfully
Thumb `uploads/vi/rMTNy_Ox4k8.jpg` created successfully from smaller
Thumb `uploads/vi/CmBha60urgc.jpg` ALREADY EXISTS
Thumb `uploads/vi/7KhzTauoP1Q.jpg` created successfully
Thumb `uploads/vi/tEw_tH-7dQU.jpg` created successfully
Thumb `uploads/vi/H7NY3hd1PL8.jpg` created successfully from smaller
Thumb `uploads/vi/ZsZd4wrlbic.jpg` created successfully from smaller

4. Further Future Advancements

Additionally, I could have saved somehow the titles of the videos to display them, and I could have also made a script to download and archive the video files themselves, but I would have just turned myself into a mirror of Youtube, which I have no desire or financial ability to do. However, using the trim($line) variable in my PHP script, I could theoretically make something like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

foreach(getDirContents($path) as $filename) {
	if(substr($filename, -3) =='.md') {
		// echo $filename."\n";
		$handle = fopen($filename, "r");
		while (($line = fgets($handle)) !== false) {
			if (strpos($line, 'youtube') !== false) {
				$line = str_replace('{{ < youtube id="', "", $line);
				$line = str_replace('{{ < youtube ', "", $line);
				if( (trim($line) != '') && (trim($line) != '-') ) {
					$output .= trim($line)."\n";
				}
			}						
		}
	}
}
$file = './ytmirror/ytmirror.txt';
file_put_contents($file, $output);
echo "\n=======================\nNow run:\n";
echo "cd ytmirror/\n";
echo "yt-dlp --id -a ytmirror.txt\n";

But I wouldn’t bother doing such thing.

5. Displaying the Embed

And that’s it, now all embeds display now enough stuff to be informative, without sending any of your requests to Google. The whole shebang took me about an hour and a half to set up, run and sync to the server and at the moment the images are taking only about 155 MB of data on the server, so that’s not a lot. Feel free to make the code above better and use it wherever and however you want!

youtube • video ID: e8Uhf3gM1m0

click image to load player

Update

PS: Upon writing this article, it seems that out of the 2169 youtube embeds I added to my posts over the years, at least 127 are not working anymore, either made private by their uploaders or removed by youtube entirely. Most likely those videos are lost forever.