Home » Mailing list archives » freebsd-hackers » 2010 » November » 8

Re: ccache pausing in buildworld

Previous thread: Space character in rc.conf variable by Harald Servat on Saturday, October 30, 2010 - 11:56 am. (6 messages)

Next thread: ABI compatibility checker for shared libraries by Gleb Kurtsou on Monday, November 1, 2010 - 6:20 am. (1 message)
[view original message]
From: Warren Block
Subject: ccache pausing in buildworld
Date: Sunday, October 31, 2010 - 9:38 am

About a month ago, ccache began to pause in buildworld.  The build 
doesn't halt or quit, it stays running but not doing anything:

/usr/local/libexec/ccache/world-cc -fpic -DPIC -O2 -pipe -march=prescott -I/usr/src/lib/libc/include -I/usr/src/lib/libc/../../include -I/usr/src/lib/libc/i386-DNLS -D__DBINTERFACE_PRIVATE -I/usr/src/lib/libc/../../contrib/gdtoa -DINET6 -I/usr/obj/usr/src/lib/libc -I/usr/src/lib/libc/resolv -D_ACL_PRIVATE -DPOSIX_MISTAKE -I/usr/src/lib/libc/locale -DBROKEN_DES -DPORTMAP -DDES_BUILTIN -I/usr/src/lib/libc/rpc -DYP -DNS_CACHING -DSYMBOL_VERSIONING -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -c crypt_xdr.c -o crypt_xdr.So

And there it stays:

   load: 0.02  cmd: make 83143 [select] 401.32r 0.05u 0.05s 0% 852k
   load: 0.01  cmd: make 83143 [select] 409.08r 0.05u 0.05s 0% 852k
   load: 0.01  cmd: make 83143 [select] 422.00r 0.05u 0.05s 0% 852k

The file where it pauses varies depending on the number of jobs (-j) 
option.  The example above is with -j6; -j1 doesn't fix it, although it 
pauses on nslexer.c instead.

This is on 8-stable as of today, i386.  The -march=prescott option comes 
from CPUTYPE?=core2 in make.conf, and removing that setting doesn't fix 
the problem.

buildworld without ccache works fine, just takes more than twice as 
long.

The kernel target works fine with or without ccache.

Any ideas?
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Garrett Cooper
Subject: Re: ccache pausing in buildworld
Date: Sunday, October 31, 2010 - 4:05 pm

Have you tried trussing or ktrace'ing the processes to determine
where they get stuck?
Thanks,
-Garrett
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Warren Block
Subject: Re: ccache pausing in buildworld
Date: Sunday, October 31, 2010 - 5:21 pm

Not until now:

# truss -p 89904
wait4(0xffffffff,0xbfbfdda8,0x3,0x0,0x0,0x0)	 = 0 (0x0)
select(1024,{4},0x0,0x0,{2.000000 })		 = 0 (0x0)
wait4(0xffffffff,0xbfbfdda8,0x3,0x0,0x0,0x0)	 = 0 (0x0)
select(1024,{4},0x0,0x0,{2.000000 })		 = 0 (0x0)

Like it's waiting for input.  Yet do it alone, and it works fine:

   # cd /usr/obj/usr/src/lib/libc
   # /usr/local/libexec/ccache/world-cc -O2 -pipe -march=prescott -I/usr/src/lib/libc/include -I/usr/src/lib/libc/../../include -I/usr/src/lib/libc/i386 -DNLS -D__DBINTERFACE_PRIVATE -I/usr/src/lib/libc/../../contrib/gdtoa -DINET6 -I/usr/obj/usr/src/lib/libc -I/usr/src/lib/libc/resolv -D_ACL_PRIVATE -DPOSIX_MISTAKE -I/usr/src/lib/libc/locale -DBROKEN_DES -DPORTMAP -DDES_BUILTIN -I/usr/src/lib/libc/rpc -DYP -DNS_CACHING -DSYMBOL_VERSIONING -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -c nslexer.c
   #

Emanuel Haupt points out ports/151287, which is the same thing.
[ message continues ]
[view original message]
From: Garrett Cooper
Subject: Re: ccache pausing in buildworld
Date: Sunday, October 31, 2010 - 6:02 pm

Question is, is it ccache's fault, or something else's?
    FWIW, it would be really nice to see what the 4th file descriptor
actually maps to, and what command is being run at the time, as well
as what other commands are being run.
Thanks,
-Garrett
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Adam Vande More
Subject: Re: ccache pausing in buildworld
Date: Sunday, October 31, 2010 - 7:31 pm

The error doesn't seem to appear when ccache is disabled.   Another data
point is that the is an error spit out when ccache is run with a clean
cache.

w/ -j8:

/usr/local/libexec/ccache/world-cc -O2 -fno-strict-aliasing -pipe
-I/usr/src/lib/libc/include -I/usr/src/lib/libc/../../include
-I/usr/src/lib/libc/i386 -DNLS -D__DBINTERFACE_PRIVATE
-I/usr/src/lib/libc/../../contrib/gdtoa -DINET6
-I/usr/obj/lib32/usr/src/lib/libc -I/usr/src/lib/libc/resolv -D_ACL_PRIVATE
-DPOSIX_MISTAKE -I/usr/src/lib/libc/locale -DBROKEN_DES -DPORTMAP
-DDES_BUILTIN -I/usr/src/lib/libc/rpc -DYP -DNS_CACHING -DSYMBOL_VERSIONING
-std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k
-Wno-uninitialized -Wno-pointer-sign -c fork.S
/usr/local/libexec/ccache/world-cc -DPROF -O2 -fno-strict-aliasing -pipe
-I/usr/src/lib/libc/include -I/usr/src/lib/libc/../../include
-I/usr/src/lib/libc/i386 -DNLS -D__DBINTERFACE_PRIVATE
-I/usr/src/lib/libc/../../contrib/gdtoa -DINET6
-I/usr/obj/lib32/usr/src/lib/libc -I/usr/src/lib/libc/resolv -D_ACL_PRIVATE
-DPOSIX_MISTAKE -I/usr/src/lib/libc/locale -DBROKEN_DES -DPORTMAP
-DDES_BUILTIN -I/usr/src/lib/libc/rpc -DYP -DNS_CACHING -DSYMBOL_VERSIONING
-std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k
-Wno-uninitialized -Wno-pointer-sign -c fork.S -o fork.po
/usr/local/libexec/ccache/world-cc -fpic -DPIC -O2 -fno-strict-aliasing
-pipe -I/usr/src/lib/libc/include -I/usr/src/lib/libc/../../include
-I/usr/src/lib/libc/i386 -DNLS -D__DBINTERFACE_PRIVATE
-I/usr/src/lib/libc/../../contrib/gdtoa -DINET6
-I/usr/obj/lib32/usr/src/lib/libc -I/usr/src/lib/libc/resolv -D_ACL_PRIVATE
-DPOSIX_MISTAKE -I/usr/src/lib/libc/locale -DBROKEN_DES -DPORTMAP
-DDES_BUILTIN -I/usr/src/lib/libc/rpc -DYP -DNS_CACHING -DSYMBOL_VERSIONING
-std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k
-Wno-uninitialized -Wno-pointer-sign -c fork.S -o fork.So
/usr/local/libexec/ccache/world-cc -O2 -fno-strict-aliasing ...
[ message continues ]
[view original message]
From: Emanuel Haupt
Subject: Re: ccache pausing in buildworld
Date: Monday, November 8, 2010 - 6:48 am

I've just updated the ccache port to 3.1.1.

Excerpt from the ChangeLog:
  
    * ccache now falls back to preprocessor mode when a non-regular include file
      (device, socket, etc) has been detected so that potential hanging due to
      blocking reads is avoided.
    * CRC errors are now detected when decompressing compressed files in the
      cache.
    * Fixed potential object file corruption race on NFS.
    * Minor documentation corrections.
    * Fixed configure detection of ar.

After updating and zeroing the cache files the a build of RELENG_8_1 fails with:

CC='/usr/local/libexec/ccache/world-cc' mkdep -f .depend -a    -I/usr/src/lib/csu/i386-elf/../common -I/usr/src/lib/csu/i386-elf/../../libc/include /usr/src/lib/csu/i386-elf/crti.S /usr/src/lib/csu/i386-elf/crtn.S
/usr/local/libexec/ccache/world-cc -O2 -pipe -I/usr/src/lib/csu/i386-elf/../common  -I/usr/src/lib/csu/i386-elf/../../libc/include -std=gnu99  -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wcast-align -Wunused-parameter -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wno-pointer-sign -c /usr/src/lib/csu/i386-elf/crti.S
/usr/local/libexec/ccache/world-cc -O2 -pipe -I/usr/src/lib/csu/i386-elf/../common  -I/usr/src/lib/csu/i386-elf/../../libc/include -std=gnu99  -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wcast-align -Wunused-parameter -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wno-pointer-sign -c /usr/src/lib/csu/i386-elf/crtn.S
/usr/local/libexec/ccache/world-cc -O2 -pipe -I/usr/src/lib/csu/i386-elf/../common  -I/usr/src/lib/csu/i386-elf/../../libc/include -std=gnu99  -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith ...
[ message continues ]
[view original message]
From: Alexey Shuvaev
Subject: Re: ccache pausing in buildworld
Date: Monday, November 8, 2010 - 8:34 am

Just FYI, you would see exact the same problem if you try to compile
i386 world on amd64-CURRENT with clang:
...
/src/lib/csu/i386-elf/crt1_s.S: Assembler messages:
/src/lib/csu/i386-elf/crt1_s.S:34: Error: suffix or operands invalid for `push'
/src/lib/csu/i386-elf/crt1_s.S:37: Error: `8(%ebp)' is not a valid 64 bit base/index expression
/src/lib/csu/i386-elf/crt1_s.S:38: Error: suffix or operands invalid for `push'
/src/lib/csu/i386-elf/crt1_s.S:39: Error: `4(%ebp)' is not a valid 64 bit base/index expression
/src/lib/csu/i386-elf/crt1_s.S:40: Error: suffix or operands invalid for `push'
clang: error: assembler command failed with exit code 1 (use -v to see invocation)
*** Error code 1
...

As far as I understand this, there is no clang cross-compiler support
at the moment. So compiler generates (somehow) i386 assembler code
which amd64 assembler does not like. Looks like something similar happens
here too...

HTH,
Alexey.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Alexey Shuvaev
Subject: Re: ccache pausing in buildworld
Date: Monday, November 8, 2010 - 11:13 am

Certainly, I have all these in the host machine and native buildworld
works as expected. What I was talking about is cross-compiling, i.e.:

make TARGET=i386 buildworld

on amd64 host.

BTW, the absence of clang testers on arm architecture would not change
unless the cross-compiling issue with clang is resolved.
Cross-compilation is a MUST HAVE here (there is no other way to bootstrap
the platform).

Alexey.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Dimitry Andric
Subject: Re: ccache pausing in buildworld
Date: Monday, November 8, 2010 - 11:01 am

Have you read http://wiki.freebsd.org/BuildingFreeBSDWithClang, which
suggests to put:

.if !defined(CC) || ${CC} == "cc"
CC=clang
.endif
.if !defined(CXX) || ${CXX} == "c++"
CXX=clang++
.endif
# Don't die on warnings
NO_WERROR=
WERROR=

in your /etc/src.conf (or make.conf, depending on your preference)?

If you use that, it should work, at least for the build32 stage of
buildworld on amd64.  This workaround is only needed on amd64, not for
i386.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Emanuel Haupt
Subject: Re: ccache pausing in buildworld
Date: Monday, November 8, 2010 - 11:42 am

You must have misunderstood the issue. This thread is entirely about
ccache. I'm having no problems building w/o ccache.

Emanuel
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Emanuel Haupt
Subject: Re: ccache pausing in buildworld
Date: Monday, November 8, 2010 - 11:50 am

Sorry, I just noticed the discussion has advanced to a clang issue. My
bad.

Emanuel


_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Dimitry Andric
Subject: Re: ccache pausing in buildworld
Date: Monday, November 8, 2010 - 12:04 pm

This is during the 32-bit build stage on amd64?  If so, what will
happen, depends on how you defined CC.  Makefile.inc1 defines CC as an
environment variable for a sub-make, using something like:

	CC="${CC} -m32 ${LIB32CPUFLAGS} -DCOMPAT_32BIT \
		-isystem ${LIB32TMP}/usr/include/ \
		-L${LIB32TMP}/usr/lib32 \
		-B${LIB32TMP}/usr/lib32"

However, if you either put CC=/usr/local/libexec/ccache/world-cc on the
make command line, or in /etc/src.conf or /etc/make.conf as an
unconditional assignment, it will *always* override the above setting
through the environment, causing the 32-bit build stage to fail.

If this is the case, you could use the same construction I have been
using to get clang to build the 32-bit stage properly, e.g. put:

.if !defined(CC) || ${CC} == "cc"
CC=/usr/local/libexec/ccache/world-cc
.endif
.if !defined(CXX) || ${CXX} == "c++"
CXX=/usr/local/libexec/ccache/world-cc
.endif

in your src.conf or make.conf, as a workaround.

Really fixing this issue requires rather ugly hackery in Makefile.inc1,
bsd.*.mk and so on, though... :-/
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Warren Block
Subject: Re: ccache pausing in buildworld
Date: Monday, November 8, 2010 - 5:10 pm

buildworld on my i386 system paused in the same place as before... until 
I remembered to clear the cache.

After that, buildworld went fine and completed.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

[ message continues ]
Previous thread: Space character in rc.conf variable by Harald Servat on Saturday, October 30, 2010 - 11:56 am. (6 messages)

Next thread: ABI compatibility checker for shared libraries by Gleb Kurtsou on Monday, November 1, 2010 - 6:20 am. (1 message)