App-V 5.0 SP2 HF5 on Windows 10

Already tried using App-V 5.0 SP2 HF5 on Windows 10 Technical Preview or are you just before trying it? If yes, make sure you have at least installed Build 9860. There is a known issue with junctions. App-V 5.0 uses junctions, i.e. in “C:\ProgramData\Microsoft\AppV\Client\Integration”. You will get an error message when launch an App-V package thru the shortcut in the start menu because the file that is part of a junction folder cannot be found.

Other than that, App-V 5.0 SP2 HF5 worked perfectly on Windows 10, I haven’t experienced any issues so far with my packages….

App-v app knows its real path

Ever wondered what is being returned if a virtualized app in App-V 5.0 asks for its own path? I tested it with a simple C++ exe that I created for that particular case. It uses the GetModuleFileName API to get its own path on the system. When running the exe, the output is its path and the directory listing of the folder it’s located in.

I created an App-V 5.0 package with the exe twice in it, once in the VFS and once in the PVAD path (PVAD = c:\getPathOfExe):

– c:\getPathOfExe\getPathOfExe.exe (PVAD)
– c:\Program Files\getPathOfExe\getPathOfExe.exe (VFS)

There is nothing else in the App-V package than these 2 files.

So to finally test the behavior after having added and published the package, I opened a cmd.exe within its virtual environment and did run both of the exe files.

Running getPathOfExe.exe in PVAD:
getPathOfExe-PVAD

Right after running the exe, you see the output of the exe’s path itself. Although it has been installed to c:\getpathofexe\, it knows that the real location is in c:\programdata\app-v\…

The next output is the directory listing of the folder it’s located in. It automatically merges with the local folder c:\getpathofexe\. C:\getpathofexe\hahahaha\ exists on my system, so it merges the folder absolutely correct. Additionally I created the folder “test” within the virtual environment which is also shown as it should.

Running getPathOfExe.exe in VFS:
getPathOfExe-VFS

The behavior is the same in PVAD and VFS. GetPathOfExe.exe recognizes that it’s located in c:\programdata\app-v\… and not in the VFS c:\program files\getPathofExe\.

In the App-V package I configured to merge c:\program files\getpathofexe\ with the local folder, that why you see the folder “hahaha” shown in the directory listing. “test” folder has been, like in the PVAD test before, manually created within the virtual environment. Directory listing works as expected.

What is the take away of the lesson? Files running in a virtual environment exactly know their “real” location.

 

 

 

ctxsym.citrix.com not working

To troubleshoot Citrix issues, I often open cdf traces with CDFControl, a Citrix tool. Unfortunately I discovered that http://ctxsym.citrix.com is currently not responding (time out). You need that to download the Citrix symbol files. According to http://discussions.citrix.com/topic/354885-tmf-files-from-ctxsymcitrixcom-not-downloadable/, Citrix is aware of the issue and works on bringing the server back online. Be patient….

19.09.14: Solved, it’s back working!

Adobe Reader’s Z@….tmp files cannot be deleted

Recently Microsoft provided me a private hotfix to fix a bug in the Windows Server 2012 OS (see one of my previous posts). The fix worked great, but after a day having it in production, a new issue appeared. Since we applied the hotfix, some users started to complain that a login isn’t possible anymore. It was immediately clear, this most probably is a side-effect of the applied fix. We configured our XenApp servers the way that the users’s temp folders are located on d:\temp\%sessionid%. This folder usually gets deleted when a user logs off. Additionally in the login script we check if the d:\temp\%sessionid% folder already exists on the server and if so, we try to delete it. If the folder cannot be deleted, the user will be automatically logged off during the login. You might ask you why we do that. The reason is that we didn’t want to have temporary files of a previous user in another user’s temp folder. There is not just a security reason behind, but also a potential issue that might appear if the users starts an application that wants to create a file that already exists and cannot be changed (i.e. Lotus Notes creates such issues). Therefore we decided that we want to have a clean temp folder for every user.
Knowing that, we knew that there must be something that still remains in the temp folder of a previous user that cannot be deleted. It was Adobe Reader that created Z@<something>.tmp files that were still in use by the SYSTEM process and couldn’t be deleted. Even a tool like handle.exe from Sysinternals wasn’t able to release the handle. Only a reboot of the server indeed released the handle and the file could be deleted. This brings us to the next question. Why is deleting those files not possible anymore after the hotfix? I monitored Adobe Reader and figured out that Adobe Reader still uses a Windows API that is no longer supported by Microsoft after Windows XP! This is one of the Windows API calls when I was printing some PDF pages:

CreateScalableFontResourceW ( 0, “C:\Users\michi_000\AppData\Local\Temp\acrord32_sbx\Z@SC8AF.tmp”, “C:\Users\michi_000\AppData\Local\Temp\acrord32_sbx\Z@RC89E.tmp”, NULL )

Check out http://msdn.microsoft.com/en-us/library/windows/desktop/dd183517(v=vs.85).aspx to get more information of that function. There you can find that end of client support is Windows XP and end of server support is Windows Server 2003. So again, Adobe uses an API that isn’t support since XP/2003 server! And the private hotfix has somehow changed the behavior of that API that Microsoft won’t support anymore.

Maybe you observed that the temp folder in the example above isn’t d:\temp\%sessionid%, but instead the default one. Correct! The reason is that I was even able to reproduce the issue on my Windows 8.1 with the latest Microsoft Updates. This made me thinking about updates. And indeed it now looks like that all Windows clients and Windows server OSs with the latest Windows Updates will be able to reproduce the issue! So which update is it? Honestly writing, I didn’t remove the update to proof it is this one, but it’s very obvious. Check out this security update: http://support.microsoft.com/kb/2993651/en-us

Read “Known issue 1” and you exactly read what you experience! So what does this mean? Adobe has to recode Adobe Reader and should use another API that is supported and doesn’t produce the same issue. This is not a Microsoft issue, this is an issue that has to be fixed by Adobe.

Maybe you also came across that post: http://blogs.adobe.com/dmcmahon/2011/11/08/acrobatreader-tmp-files-left-behind-in-temp-folder-after-printing/

It’s an older post, but still valid and reflects the issue. The option with the ini file still works with the latest Adobe Reader version, but this isn’t an option for me as this will have an impact on the quality of the printouts.

To sum up – Adobe has to recode Adobe Reader. The CreateScalableFontResourceW function is outdated and not supported anymore. I was able to proof that this API doesn’t release the font file anymore after it has been executed.

Kernel debugging on a VM with WinDbg

In case you need to debug the kernel on a VM running on Hyper-V, this is how you can do it with a Windows Server 2012 R2 VM Generation 2:

  1. After the VM has been created, a COM port is needed. By default, you cannot create a COM port with the Hyper-V Manager UI. That’s one of the differences between Gen1 and Gen2. You have to use Powershell to get your COM port. First of all shut down the VM and disable SecureBoot (replace YourVMName with the name of your VM and make sure you run all commands with “run as administrator”):
    Set-VMFirmware -VMName YourVMName -EnableSecureBoot Off
  2. Then create the COM port as follow:
    Set-VMComPort -VMName YourVMName 1 \\.\pipe\DebugIT
    The “1” is the COM port you want to use (feel free to adjust it if needed) and the pipe path. The last string of the pipe can as well be adjusted, you could use something else than DebuIT.
  3. Start the VM
  4. Enable debugging:
    bcdedit /debug on
    bcdedit /dbgsettings serial debugport:n baudrate:115200
    where n is the port number defined in step 2.Reboot the VM.
  5. Open up WinDbg and choose File –> Kernel Debug
  6. Configure the COM connection according to the screen shot:
    windbg_localdebug
    Port: \\.\pipe\DebugIT
  7. You will see “Waiting to reconnect…” afterwards. That’s ok because we haven’t started yet debugging. Choose Debug –> Break (or Ctrl+Break). That’s it, you’re now able to debug the kernel of the running VM:windbg_localdebug-connected

 

Full memory dump with Windows Error Reporting

Windows Error Reporting is a nice tool from Microsoft. It collects required Information from an exception that happend and (if you want so) checks online if there is a solution for your issue. By default it creates a memory minidump of that process that crashed. Often a minidump is good enough to narrow down the issue, but what if it’s not enough? There’s the possibility to enable full Memory dumps when an application crashes. You can enable it with the following key:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps]
“DumpType”=dword:00000002

That’s it. There is no restart required.

Note that the above is a global setting (per computer). It’s also possible to set it just for 1 application. See http://msdn.microsoft.com/en-us/library/windows/desktop/bb787181(v=vs.85).aspx for more Information.

BSOD SESSION_HAS_VALID_POOL_ON_EXIT (ab)

Currently we get SESSION_HAS_VALID_POOL_ON_EXIT (ab) BSODs regularly on our Windows Server 2012 servers. These servers are used as XenApp servers and therefore a BSOD on such a server is pain in the ass when you have 50 users or even more working on such a server.

Finally Microsoft confirmed that this is a bug and they are now working on a fix. I will update this post as soon as I get a hotfix that resolves the issue.

The windbg output looks as follow:

SESSION_HAS_VALID_POOL_ON_EXIT (ab)

SESSION_HAS_VALID_POOL_ON_EXIT (ab)

Update 05.09.14: I got a private hotfix from Microsoft that definitively fixes the issue. I informed the Microsoft engineer today that they can start to initiate the process to release a public hotfix. We have seen the BSOD 1-2 times per day, and it completely disappeared since we applied the hotfix. There seems to be another bug we discovered by another BSOD that Microsoft is analyzing, but finally no AB BSOD!

Update 11.02.15: The fix has now been released for public:
More Information: https://support.microsoft.com/kb/3036220
Download: http://www.microsoft.com/en-us/download/confirmation.aspx?id=45710

ICA Session Performance counters missing

In case you’re missing the ICA Session performance counters in perfmon, run that command for XenApp/XenDesktop 7.x:

regsvr32 C:\Program files (x86)\Citrix\System32\icaperf.dll

Change the path of the dll according to your install folder, the one above is default. This will register the dll that brings back the counters:

ICASession-PerfCounters

I’m not sure why they suddenly disappeared on my servers, but that might be a side-effect from a Citrix Hotfix.
The performance counters are very useful when troubleshooting issues, especially on WAN connections. You can monitor the network latency and bandwidth used from your XenApp server to all the end clients of the users connected to the server. Also HDX Monitor gets the data from these counters.

Update 08.09.2014: Citrix provided a hotfix that brings the performance counters back: http://support.citrix.com/article/CTX141330