Thanks to Tom, I now understand I should have specified the width of the first format when I nested it in the second format. In other words, I changed [longfmt.] to [longfmt52.]:

PROC FORMAT;
VALUE longfmt
    1 = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
VALUE nestfmt
    1 = [longfmt52.];
QUIT;

PROC SQL;
CREATE TABLE tbl (col NUM);
INSERT INTO tbl VALUES (1);

SELECT col FORMAT=longfmt. FROM tbl;
SELECT col FORMAT=nestfmt. FROM tbl;
QUIT;