The SCP protocol is largely undocumentated, which is why you’re left in the dark if you’re trying to write your own implementation. However, by experimenting, I found some of the inner workings, and also how to write your own client and server.
First of all you might want to think about whether you really want to make your own implementation, or whether the -S argument to change the transport layer is maybe sufficient already. The transport program must use stdin and stdout to communicate with scp. For detailed information, run scp -v -S echo 0.0.0.0:file.in file.out and scp -v -S echo file.in 0.0.0.0:file.out. Please make sure that you don’t accidentally print log messages or similar to stdout, as this will cause the process to fail.
If you decide you want to dig deeper, there are two undocumented scp flags. The first is -f, which reads from disk into stdout, and the other is -t, which writes to disk from stdin.
As a first step, the party which receives the file (scp if -t is set, or the user if -f is set) sends a null character to communicate that it’s ready for metadata. Then, the sender sends the following line terminated with a (apparently Unix) newline:
1
C<permissions> <size> <filename>
where <permissions> is the 4-digit numeric permission code (see), <size> is the size of the file content in bytes (decimal ASCII), and <filename> is the name of the file on the server. After this, the receiving end sends another null character to confirm, and the sender sends the binary file content terminated by a null character. After everything is done, the receiver sends one last confirming null character. Note that there is exactly one newline between <filename> and the file content (there may not be a terminating newline). Other occurences of null characters are not escaped (since the file size is given, scp knows which null character is the terminating one).
Now that we know how these two flags work, the rest of scp is trivial; when you connect to a server, it starts ssh (or whatever other transport program you selected) and launches scp on the remote machine. It then uses stdin/stdout to communicate with the remote and write the file locally.
Here’s a demonstration using some mock bash scripts. readmock.sh:
1
2
3
4
5
6
#!/bin/bash
sleep 5 # make sure null character has been received
echo -ne "C0644 5 test.txt\n" # output metadata
sleep 5 # make sure another null character has been received
echo -ne "BR\0KE\0" # output file content
And writemock.sh:
1
2
3
4
5
6
7
8
9
#!/bin/bash
echo -ne "\0" # send null character
sleep 5 # make sure metadata has been sent
echo -ne "\0" # send another null character
sleep 5 # give scp time to send file
cat > protocol-info.txt # log stdin into protocol-info.txt
# note that this process must be killed with ctrl+c, since cat doesn't stop by itself
After chmod +x readmock.sh writemock.sh, you can run scp -v -S ./readmock.sh fake.host:fakefile file.txt and scp -v -S ./writemock.sh file.txt fake.host:fakefile.
Interestingly, SCP does not check the address before passing it on to the transport command; meaning we can make scp work with SCION addresses easily.