Monday, August 4, 2014

Quick Start

The Parallel Video Processor is a utility that lets the user create a cluster of PCs whose job is to encode a video.

Its hard requirements are: FFMPEG, FFProbe, and an x265 or x264 encoder somewhere on your system.
Avisynth is a soft requirement; if you have it installed and the video to be encoded is an Avisynth script, then PvP instead of relying on FFMPEG or the encoder to seek and cut the video, it will do so through Avisynth's Trim function.

In Parallel Video Processor, there is a unique master node in the cluster which issues all necessary commands to the slaves.

Let's begin by setting up a cluster.

A cluster can have only one master and at least one slave. The master can only begin after all slaves are up and running so that the proper commands can be issued, so let's start with the slave first:


And this is the slave screen:


"Load Settings" is useful when you have configured a slave before and saved the settings. Since this is our first time, there's not much to save. Let's go ahead and press 2 to configure this particular slave node.


The process is pretty easy. You choose and type your username and password, verify it and finally choose the port that PvP will listen to for incoming connections. The default port is 5445, but here we've chosen 4554.

After this is done, we're back at the original screen. We'll press 3 to show PvP where to find the files necessary to process our video.


After a few steps, PvP will try to automatically detect the video's settings. Please review carefully. In most cases PvP will get everything right, but sometimes it will not. If all (or most) settings are correct, just accept them.

 
In most cases, it is quicker to just accept these settings, as they can be tweaked through the "Configure video settings" screen.


Finally, PvP will need to know if your output directory that your slave node will write to is on some sort of a shared storage with the master node. If the master's output directory is shared with the slave's then please answer 'y'.


After we're all done, we are back at the original screen. It's time to press 4 change any video settings. If you are absolutely sure that all automatically detected settings of your video were detected correctly you can skip this step.


Note that <ENTER> autocompletes previously-entered settings (if there are any, they will be shown inside brackets), and <TAB> autocompletes if there is a list of settings available. There is an extra option for the color format, which is --bypass. Using it will not specify a color format.

Once again we are back at the original screen. Let's press 6 for our slave node to know how it can connect to our master server. This screen is similar to the slave configuration screen, only that we have to enter the master node's username and password instead of ours. We also need to enter the master server's IP and port, and not just the port.


Back at the original screen, we have all the information required to start the slave. Pressing '7' will start sanity checks and after all tests have been passed, the slave node will be listening for incoming connections. It will only accept ones from the IP we said was the master node and none else.


Time to fire up the server.

 
The configuration options available are:


Let's press '2' to let our server know how to connect to our slave nodes. This screen is similar to the 'Configure master node settings' in our slave. We'll have to enter the slave node's IP address, the port that it is listening to and its username and password.


After we're done, we need to press the <ESC> button to let PvP know that we've finished configuring slave nodes.

After the slave node configuration is finished, we are presented with the node weight configuration screen. We'll need to choose a weighting method for our nodes. The more weight a node has, the more processing will be assigned to it. In other words, a node that is twice as fast as another one, ideally should get twice as much weight; that way the nodes will finish at the same time and no time will be wasted.


There are 4 different types of weighting methods:
1. Pre-Set: Video files have already been segmented and distributed per node. Let's say that a buddy of yours has the last half of a show and you have the first. The master node will process its part, the slave accordingly and after all is finished, the final result will be one video.
2. Manual: Video files are the same across nodes and the user is asked to individually assign a weight per node (note that in this case the server will perform checksums to validate that each node has the same file).
3. Equal: Video files are the same across nodes and all nodes are assigned the same weight (note that in this case the server will perform checksums to validate that each node has the same file).
4. Benchmark: Video files are the same across nodes and the server requests all nodes to be benchmarked for the file in question, acquiring the node weights through that process (note that the server will perform checksums to validate that each node has the same file).

After you've selected your desired weighting method, you'll be back at the original screen. If you've selected to process your video file benchmarking the systems, this might be a good time to actually press '8' instead of working through the flow. This is because a benchmark might take a while. If you request your slave nodes to be benchmarked while the master node is being configured, you save time.

Back at the original screen, we need to go through the flow just like we did on our slave node. This means that next up is '3', 'Configure paths'.


The only differences between master and slave path configuration screens are these:
a) If an x264 encoder is detected, then PvP will ask you if you want to encode directly using x264 or if you want to pipe a transcoded, raw video format of your video to the x264 encoder using FFMPEG. Even though x264 can handle most input formats, piping through FFMPEG can be a more stable method of encoding a video.
b) You will be asked of an output filename and not just the output directory.

Back at the original screen, we need to go through the flow just like we did on our slave node. This means that next up is '4', 'Configure video settings'. Master screen in exactly the same as slave's.

After that, we can configure any encoder settings we want by pressing '5'. We can choose to enter any x264 or x265 commands except those that are handled by PvP. You can enter x264 commands even though you are using x265 and vice-versa. This is handy because if you save your settings and decide to use a different encoder, you don't have to go through the screen again.


As with other screens, <TAB> autocompletes when there's a list of options available.

Back at the original screen, it's time to configure our master node.


This screen is similar to the slave configuration screen we did on the slave node. Of course we need to put the master node's credentials and port.

Looks like we're ready to go. After we press '7' and a few sanity checks, all nodes should start encoding.

Slave:



Master:
 
After the encoding's finished and everything's gone well, you'll be asked if you want to delete the now useless temporary files that PvP has created and whether you want to issue a shutdown of the slave nodes' networking components for security reasons.

You've just created a cluster, encoded a video using a state-of-the-art encoder and cut the processing time required to almost half. Congrats!

No comments:

Post a Comment