Yes, by making RetVal a BindableProperty and setting up the appropriate bindings, you can bind it to the ViewModel's property and set it properly before the popup is closed.
Note: Make sure that your ViewModel implements INotifyPropertyChanged to support data binding, and that the RetVal property has get and set accessors with proper notification.