Previously if we use the in-memory backend, it will return a raw
binary token for refresh token. This fixes the case.