I finally found out

Map<Object, Object> studentMap2 = studentList
                .stream()
                .flatMap(t -> t.getSubjectList().stream())
                .toList()
                .stream()
                .collect(Collectors.groupingBy(t ->t.getSubjectName()
                            ,Collectors.collectingAndThen(
                                    Collectors.toList(),
                                        t -> t.stream().sorted(Comparator.comparingDouble(Subject::getMarks ).reversed())
                                        .limit(3)
                                        .collect(Collectors.toList())
                                        )));
System.out.println(studentMap2);

Below is the result which i get

{English=[Subject [studentId=1007, subjectName=English, marks=92.0]
        , Subject [studentId=1002, subjectName=English, marks=83.0]
        , Subject [studentId=1001, subjectName=English, marks=72.0]], 
Maths=[Subject [studentId=1007, subjectName=Maths, marks=98.0],
       Subject [studentId=1003, subjectName=Maths, marks=91.0],
       Subject [studentId=1001, subjectName=Maths, marks=87.0]]}

Thanks @AliRaza Your answer helped.